Manipulating a test based on a subset of similar divergent routes from different organizations

ABSTRACT

System, method, and non-transitory medium for utilizing routes followed by runs of test scenarios to manipulate a test scenario template. Runs of test scenarios run by users belonging to different organizations on software systems associated with the different organizations are analyzed to identify routes that the runs follows; the identified routes may be stored in a database. A route retriever receives a certain template route belonging to a test scenario template and retrieves from the database divergent routes that diverge from the template route. A subset selector selects from the divergent routes a subset of divergent routes that are similar to each other. A template manipulator manipulates the test scenario template according to the subset if the size of the subset reaches a predetermined threshold.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a Continuation-In-Part of application Ser. No.13/103,078, filed May 8, 2011. This Application claims the benefit ofU.S. Provisional Patent Application No. 61/747,313, filed Dec. 30, 2012,and U.S. Provisional Patent Application No. 61/814,305, filed Apr. 21,2013. This application also claims the benefit of U.S. ProvisionalPatent Application No. 61/919,773, filed Dec. 22, 2013, the entirecontents of which is herein incorporated by reference in its entiretyfor all that it teaches without exclusion of any part thereof.

The following co-pending US Patent Applications, filed on Dec. 27, 2013:Ser. Nos. 14/141,514; 14/141,564; 14/141,588; 14/141,623; 14/141,655;14/141,676; 14/141,726; 14/141,859; 14/141,887; 14/141,925; 14/141,974,and the following co-pending US Patent Applications, filed on Dec. 28,2013: Ser. Nos. 14/142,768; 14/142,770; 14/142,771; 14/142,772;14/142,774; 14/142,781; 14/142,783; 14/142,784, may include relatedsubject matter.

BACKGROUND

Software systems, such as Enterprise Resource Planning (ERP) systems,are often used by organizations to perform the numerous and diversecomputational tasks involved in conducting business. The softwaresystems may involve many different software modules, which may also becustomized to meet a specific organization's needs. Each time a moduleis installed, updated, and/or customized, it typically needs to betested in order to verify that it operates as expected. Organizationsoften devote many resources for the purpose of testing and validatingthe performance of their software systems.

However, creating tests is not a trivial task. It is not always simpleto determine what elements should be tested (e.g., which softwaremodules, business processes, and/or transactions), or how conduct thetesting of those elements. Building an effective and relevant testingsuite is usually a time-consuming process, built on trial and error anddomain know-how. Until an effective testing suite is built, andorganization may end up conducting inefficient and/or incomplete testingof its systems.

One hurdle faced by test designers is the vast diversity of systemoptions and system behavior that may be encountered and need to betested. For example, business processes may have different behaviors tocover different options, different users, and\or for special cases.Generating tests that cover all cases is not trivial; it may requireforesight, imagination, and experience that may not be available to thetest designer. For example, the test designer may not be aware ofcertain options that are available in the system since they were neverutilized by the organization to which the test designer belongs;however, a recent update to the system has caused these options to beapplicable for the organization. In this case, the test designer may notanticipate, nor have knowledge of how to test, the new options.

It is often the case that software systems belonging to differentorganizations utilize many software modules that are the same orsimilar, and/or software modules that involve similar customizations.Consequently, testers belonging to the different organizations often endup running tests for testing those same or similar components. Thus,were organizations able to utilize each other's testing-relatedknowledge, they might be able to expand their testing arsenal and comeup with more effective, comprehensive, and relevant tests. For example,after an update to a certain software module, a certain organization maydevise a few tests to verify the system's performance. The testdesigners at the certain organization may have overlooked certainaspects of the module, which they did not include in the tests. However,test designers in other organizations may have identified those aspectsand addressed them in the tests they devised. Were there sharing ofinformation between organizations, the tests devised for the certainorganization might have ended up including the overlooked aspects.

BRIEF SUMMARY

Some aspects of this disclosure involve methods, systems, and/ornon-transitory computer-readable medium, which enable manipulation of atest scenario template in order to update it, or create a new template.Optionally, the manipulation is performed according to a subset ofdivergent routes that are selected from among routes retrieved from adatabase. Optionally, a route defines a sequence of one or more elementsinvolved in running a test scenario, such as test steps, transactions,and/or screens. The routes retrieved from the database are similar to acertain template route which belongs to the test scenario template.Thus, the manipulation according to divergent routes may considered aform of testing data sharing among organizations. The sharing may alloworganizations running certain tests to obtain information that mayenable them to create additional tests involving additional aspects notaddressed by their original tests.

In some embodiments, a test scenario template is a model according towhich a test scenario may be structured. A test scenario template mayinclude one or more test steps, which instruct a user on an action to beperformed as part of the test, such as where to enter a value, whatbutton to push, what screen to select, or what transaction to run.Optionally, a test scenario template may include one or more defaultvalues used for running an instantiation of the test scenario template(e.g., default values for certain fields in a screen).

One aspect of this disclosure involves a computer system that isconfigured to utilize routes followed by runs of test scenarios tomanipulate a test scenario template. The computer system includes a testidentifier that is configured to identify runs of test scenarios run byusers belonging to different organizations on software systemsassociated with the different organizations. Additionally, the computersystem includes a route analyzer, a database, and a route retriever. Theroute analyzer is configured to receive a run of a test scenario and toidentify a route that the run follows. The database is configured tostore the route. The route retriever is configured to receive a certaintemplate route belonging to a test scenario template and to retrievefrom the database divergent routes that diverge from the template route.Optionally, the test scenario template is associated with a certainorganization that does not belong to the different organizations. Thecomputer system also includes a subset selector and a templategenerator. The subset selector is configured to select from thedivergent routes a subset of divergent routes that are similar to eachother, and the template manipulator is configured to manipulate the testscenario template according to the subset if the size of the subsetreaches a predetermined threshold. Optionally, the template manipulatoris configured to manipulate the test scenario template by updating thetest scenario template according to one or more of the routes belongingto the subset. Additionally or alternatively, the template manipulatormay be configured to manipulate the test scenario template by generatinga new test scenario template based on the test scenario template and oneor of the routes belonging to the subset.

In one embodiment, the computer system optionally includes a datacleaner that is configured to select a value from the manipulated testscenario template, and remove the selected value from the manipulatedtest scenario template if the selected value does not appear in runs oftest scenarios that follow at least two routes that belong to thesubset. In another embodiment, the computer system optionally includes amonitoring module that is configured to monitor the users running thetest scenarios on the software systems that belong to the differentorganizations and to provide data obtained from the monitoring to thetest identifier. In yet another embodiment, the computer systemoptionally includes a user interface that is configured to suggest to auser to run an instantiation of the manipulated test scenario template.

Another aspect of this disclosure involves a computer implemented methodfor utilizing routes followed by runs of test scenarios to manipulate atest scenario template. Executing the method may involve performing thefollowing: Identifying runs of test scenarios run by users belonging todifferent organizations on software systems associated with thedifferent organizations. Receiving a certain run of a test scenario.Identifying a route that the certain run follows. Storing the route in adatabase. Receiving a certain template route belonging to a testscenario template. Optionally, the test scenario template is associatedwith a certain organization that does not belong to the differentorganizations. Retrieving from the database divergent routes thatdiverge from the template route. Selecting from the divergent routes asubset of divergent routes that are similar to each other. Andmanipulating the test scenario template according to the subset if thesize of the subset reaches a predetermined threshold. Optionally,manipulating the test scenario template according to the subset involvesupdating the test scenario template according to one or more divergentroutes belonging to the subset. Additionally or alternatively,manipulating the test scenario template according to the subset mayinvolve generating a new test scenario template based on the testscenario template and one or more divergent routes belonging to thesubset.

In one embodiment, executing the method may optionally involvemonitoring the users running the test scenarios on the software systemsthat belong to different organizations and providing data obtained fromthe monitoring for use in the identifying of the runs of test scenarios.In another embodiment, executing the method may optionally involveselecting a value from the manipulated test scenario template, andremoving the selected value from the manipulated test scenario templateif the selected value does not appear in runs that follow at least twoof the divergent routes belonging to the subset. In yet anotherembodiment, executing the method may optionally involve suggesting to auser to run an instantiation of the manipulated test scenario template.

Yet another aspect of this disclosure involves a non-transitorycomputer-readable medium for use in a computer to utilize routesfollowed by runs of test scenarios to manipulate a test scenariotemplate. The computer includes a processor, and the non-transitorycomputer-readable medium includes the following program code: Programcode for identifying runs of test scenarios run by users belonging todifferent organizations on software systems associated with thedifferent organizations. Program code for receiving a certain run of atest scenario. Program code for identifying a route that the runfollows. Program code for storing the route in a database. Program codefor receiving a certain template route belonging to a test scenariotemplate. Optionally, the test scenario template is associated with acertain organization that does not belong to the differentorganizations. Program code for retrieving from the database divergentroutes that diverge from the template route. Program code for selectingfrom the divergent routes a subset of divergent routes that are similarto each other. And program code for manipulating the test scenariotemplate according to the subset if the size of the subset reaches apredetermined threshold.

In one embodiment, the program code for manipulating the test scenariotemplate according to the subset includes program code for updating thetest scenario template according to one or more of the divergent routesbelonging to the subset. In another embodiment, the program code formanipulating the test scenario template according to the subset includesprogram code for generating a new test scenario template based on thetest scenario template and one or more of the divergent routes belongingto the subset. In one embodiment, the program code stored in thenon-transitory computer-readable medium optionally includes program codefor selecting a value from the manipulated test scenario template, andremoving the selected value from the manipulated test scenario templateif the selected value does not appear in runs that follow at least twoof the divergent routes belonging to the subset. Optionally, the programcode also involves testing that the at least two of the divergent routesare associated with at least two different organizations.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are herein described, by way of example only, withreference to the accompanying drawings. In the drawings:

FIG. 1 illustrates one embodiment of a computer system configured toutilize runs of test scenarios run by users belonging to differentorganizations to manipulate a test scenario template according todivergent routes in the runs;

FIG. 2 illustrates one embodiment of a computer system configured toutilize runs of test scenarios run by users belonging to differentorganizations to manipulate a test scenario template according todivergent routes in the runs;

FIG. 3 illustrates updating of a test scenario template according todivergent routes;

FIG. 4 illustrates one embodiment of a computer implemented method forutilizing divergent routes identified in runs of test scenarios tomanipulate a test scenario template;

FIG. 5 illustrates one embodiment of a computer implemented method forutilizing divergent routes identified in runs of test scenarios tomanipulate a test scenario template;

FIG. 6 illustrates one embodiment of a computer system configured toutilize routes followed by runs of test scenarios to manipulate a testscenario template;

FIG. 7 illustrates one embodiment of a computer system configured toutilize routes followed by runs of test scenarios to manipulate a testscenario template;

FIG. 8 illustrates one embodiment of a computer implemented method forutilizing routes followed by runs of test scenarios to manipulate a testscenario template;

FIG. 9A illustrates a combination of system modules that may be used inembodiments described in this disclosure;

FIG. 9B illustrates a combination of system modules that may be used inembodiments described in this disclosure;

FIG. 9C illustrates a combination of system modules that may be used inembodiments described in this disclosure;

FIG. 9D illustrates a combination of system modules that may be used inembodiments described in this disclosure;

FIG. 9E illustrates a combination of system modules that may be used inembodiments described in this disclosure;

FIG. 9F illustrates a combination of system modules that may be used inembodiments described in this disclosure;

FIG. 9G illustrates a combination of system modules that may be used inembodiments described in this disclosure;

FIG. 10A illustrates a template route that includes a sequence of sevenscreens;

FIG. 10B illustrates a divergent route that diverges from the templateroute and later on converges back to the template route;

FIG. 10C illustrates a divergent route that diverges from the templateroute and does not converge back to the template route; and

FIG. 10D illustrates a divergent route that merges with the templateroute.

DETAILED DESCRIPTION

The term “transaction” is defined as a computer program, such as SAP ERPtransaction or Oracle Application Form. In one example, a transactionmay enable a user to access a certain functionality and/or may be calledby filling its code in a box in a screen and/or by selecting it from amenu. In another example, a transaction is a logical portion of work,performed by a processor, involving the execution of one or more SQLstatements.

The term “test step” refers to one or more actions performed via a UserInterface (UI) as part of running a test scenario. In some cases,performing actions via a user interface may be achieved by interactingwith the user interface, and/or by interacting with an ApplicationProgram Interface (API) related to the user interface.

The terms “a description of a run of a test scenario”, also referred toas “a run of a test scenario”, refer to data pertaining to running atest scenario on a software system (e.g., inputs, outputs, and/orintermediate data generated prior to running the test scenario, datagenerated during its run, and/or data generated as a result of runningthe test scenario). In one example, a run of test scenario may beobtained from monitoring a user running the test scenario on a softwaresystem. In the interest of brevity, in this disclosure, a term like “runof a test scenario” may be replaced with the shorter “run”, where it isclear from the context. A run of a test scenario may be referred to asbeing “run by a user”. This means that data included in the run of thetest scenario is related to, or generated from, activity of the user ona software system, in which test steps of the test scenario wereexecuted. Optionally, at least some of the data included in the run ofthe test scenario is derived from monitoring the activity of the user,which is related to execution of the test steps. Additionally, a run ofa test scenario may be referred to as being associated with anorganization, meaning that the run of the test scenario was run by auser belonging to the organization. Optionally, the user belonging tothe organization ran the test scenario, at least in part, on a softwaresystem that belongs to the organization. Moreover, if it is mentioned,for example, that runs of test scenarios are received or clustered, itis meant that the objects being received may be processed descriptionsof the runs of test scenarios (e.g., describing various attributes ofthe runs of the test scenarios), and not necessarily the actual rawrecorded data that was initially obtained from monitoring users runningthe test scenarios.

A run of a test scenario may be considered an instantiation of the testscenario. That is, a certain test scenario may be run several times. Forexample, a test scenario may be run by different users, run on differentsystems, and/or run by the same user on the same system at differenttimes. Each time the test scenario is run, that may be considered anevent of instantiating the test scenario, and each run of the testscenario may be considered an instantiation of the test scenario.

In some embodiments, runs of test scenarios are identified from dataobtained from monitoring users. Optionally, monitoring users may involvedetecting, recording, and/or analyzing information entered by the usersto computer systems and/or information presented to the users by thecomputer systems. Additionally or alternatively, monitoring may involvelogging programs that were executed by the users, values utilized by theprograms, memory content of programs and/or network traffic related toactivities taken by the users. Optionally, a run of a test scenario mayinclude data obtained from monitoring that underwent processing,summarization and/or analysis. Thus, a run of a test scenario need notnecessarily include all data obtained from monitoring the running of thetest scenario, and/or include data obtained from the monitoring in thesame form as it was originally collected in.

A run of a test scenario may include descriptions of various aspects ofrunning the test scenario such as: (i) the identity and/or composition(e.g., field types and/or identifiers) of a user interface (UI) screensthe user manipulated and/or was exposed to while running the testscenario; (ii) user interactions with a system (e.g., actions performedby the user) (iii) transactions executed; (iii) behavior of the systemduring the test scenario (e.g., content of network transmissions,procedure calls, requests made to components of the system); and/or (iv)representations of the state of the system at various stages before,during, and/or after the test scenario is run. Additionally, a run of atest scenario may include data extracted from the test scenario and/ortemplate of which the run is an instantiation. For example, the run mayinclude values taken from a script according to which the run isexecuted and/or default values that appeared in template of which therun is an instantiation. The run may even include portions, or theentire scope, of the test scenario and/or the template of which the runis an instantiation.

A test scenario that is run by a user may be characterized according toits degree of automation, depending on type of involvement required ofthe user. In cases where the user is required to enter most of thevalues (e.g., field values on a screen), the run of the test scenariomay be considered to be a manual run. In other cases, where the systemprovides some of the values (e.g., automatically fills values in no morethan 95% of the fields on a screen/transaction/business process), butthe user is still required to provide other values (e.g. enter values tofields that remain without values in the screen/transaction/businessprocess), the run of the test scenario may be considered to be asemiautomatic run. In cases where little to no user intervention isneeded (e.g., the script for running a test includes more than 95% ofthe values to be entered to the UI for the test, and is read by thesystem), the run of the test scenario may be considered to be anautomatic run.

The term “test scenario template” refers to a model according to which atest scenario may be structured. A test scenario template may includeone or more test steps, which instruct a user on an action to perform aspart of the test, such as where to enter a value, what button to push,what screen to select, or what transaction to run. Optionally, a testscenario template may include one or more default values used forrunning an instantiation of the test scenario template (e.g., defaultvalues for certain fields in a screen). Additionally or alternatively, atest scenario template may be missing one or more values that arerequired for running an instantiation of the test scenario template; insuch a case, a user running the instantiation of a template may providethe one or more missing values. In the interest of brevity, in thisdisclosure, a term like “test scenario template” may be replaced withthe shorter “template”, where it is clear from the context.

A run of a test scenario based on a test scenario template may beconsidered an instantiation of the test scenario template. For example,different users may run test scenarios based on a certain template; eachtime one of the users runs a test scenario based on the certaintemplate, the corresponding run of that test scenario is considered aninstantiation of the certain template. Similarly, if a user runsmultiple test scenarios based on a template, where each time a testscenario was run it was run on a same software system, each of the runsof the test scenarios is considered a separate instantiation of thecertain template. Optionally, a test scenario template may be consideredassociated with an organization if a user belonging to the organizationran an instantiation of the template and/or is intended to run aninstantiation of the template. Additionally or alternatively, a testscenario template may be considered associated with an organization ifthe template was generated based on one or more runs of test scenariosthat are associated with the organization.

As used herein, the term “software system” refers to a computer systemthat has software components (e.g., software modules that includeprograms). A software system may involve hardware (e.g., servers withprocessors) on which the software may run. The hardware may be dedicatedhardware for the software system (e.g., servers sitting at anorganization to which the software systems belong). Additionally oralternatively, hardware involved in a software system may be allocatedon demand (e.g., cloud-based servers that are utilized by the softwaresystem as needed by it).

FIG. 1 and FIG. 2 illustrate embodiments of a computer system configuredto utilize runs of test scenarios run by users belonging to differentorganizations to manipulate a test scenario template according todivergent routes in the runs. The illustrated embodiments include atleast a test identifier 354, a route analyzer 356, a database 358, aroute counter 360, and a template manipulator 362.

Herein, a route defines a sequence of one or more elements involved in arun of a test scenario; an element may be a test step, a transactions,or a screen. Additionally, a test scenario template may define atemplate route that instantiations of the template are supposed tofollow. Thus, a run of a test scenario based on a template is expectedto reflect the template route; for example, the run may describe screensand/or transactions included in the template route.

The test identifier 354 is configured to identify runs of test scenariosrun by users belonging to different organizations on software systemsassociated with the different organizations. Optionally, the runs areidentified based on activity data 353 obtained from monitoring theusers.

In one embodiment, the route analyzer 356 is configured to receive acertain run of a test scenario instantiated from a test scenariotemplate. Optionally, the route analyzer 356 may be configured toreceive many runs of test scenarios; the description below describesexamples of what may happen with the certain run. In one example, thecertain run was run by a user after being suggested the template via auser interface. Optionally, the certain run of the test scenario was runon a software system associated with a certain organization that doesnot belong to the different organizations. Optionally, the differentorganizations and the certain organization are associated with differentfields of operation (e.g., the different organizations are in theinsurance field, while the certain organization is in the field ofautomobile manufacture). Alternatively, the different organizations andthe certain organization are associated with a same field of operation.

The test scenario template defines a template route that instantiationsof the template are supposed to follow. That is, each run of a testscenario instantiated from the template is supposed to reflect a certainsequence of elements, as defined by the template route. Optionally, adivergent route that diverges from a template route is characterized bya different sequence of elements compared to sequence of elementsdefined by the template route.

In one embodiment, based on the elements identified in the certain run,the route analyzer 356 characterizes a route followed by the certainrun. The route analyzer 356 is also configured to identify that thecertain run follows a certain divergent route that diverges from thetemplate route. For example, the route analyzer 356 may compare elementsin the template route with elements identified in the certain run inorder to determine whether the elements are similar and/or appear in thesame order as the elements in the template route.

In one embodiment, the route analyzer 356 is configured to identifydivergent routes that diverge from the template route and later onconverge back to the template route. Additionally or alternatively, theroute analyzer 356 is also configured to identify divergent routes thatdiverge from the template route and do not converge back to the templateroute. Additionally or alternatively, the route analyzer is alsoconfigured to identify divergent routes that merge with the templateroute.

In one embodiment, if there is an essential difference between thecertain route and the template route the template route it is considereda divergent route. For example, an essential difference may be a singleelement that differs between the certain route and the template route(e.g., a screen in the certain route that is not in the template route,a transaction in the template route that is not executed when followingthe certain route). Optionally, an essential difference involves atleast a certain number of elements and/or a certain proportion ofelements. For example, there may be an essential difference between thecertain route and the template route if they differ on at least 5elements and/or 15% of the elements.

In one embodiment, the database 358 is configured to store the certaindivergent route identified by the route analyzer 356. Optionally, thedatabase 358 may be configured to store many divergent routes; theexamples listed below describe various possibilities for how thedatabase 358 may handle the certain divergent route.

In one example, each time a divergent route is identified in a run of atest scenario, the divergent route is stored in the database 358.Optionally, storing a divergent route involves storing elements includedin the divergent route, a run of a test scenario that follows thedivergent route, information pertaining to a user that ran a testscenario that followed the divergent route, and/or informationpertaining to an organization associated with a run of a test scenariothat followed the divergent route.

In another example, each time a divergent route is identified in a runof a test scenario, the route analyzer 356 and/or the database 358determine whether a route essentially the same as the divergent routehas been already stored. If no essentially same divergent route has beenstored, the divergent route is stored.

In one embodiment, routes that are essentially the same include the samesequence of screens. Optionally, routes that are essentially the samehave a same sequence of screens but with different user keys.Optionally, routes that are essentially the same have a same sequence ofscreens but utilize different default values for at least some of thefields displayed on the screens.

Optionally, if an essentially same route has already been stored once,then a certain counter associated to the divergent route is incremented.Optionally, such a counter may be a counter corresponding to runsfollowing the divergent route, a counter corresponding to users that ranruns following the divergent route, and/or a counter corresponding toorganizations associated with runs following the divergent route.

In one embodiment, the route counter 360 is configured to count numberof divergent routes stored in the database 358 that are essentially thesame as the certain divergent route. Optionally, the route counter 360queries and/or searches the database 358. Optionally, by processingresults returned by the search and/or query the route counter 360 isable to return the number of number of divergent routes that areessentially the same as the certain divergent route. For example, theroute counter 360 counts the number of divergent routes returned in thesearch and/or query. Optionally, the route counter 360 bases the numberof divergent routes that are essentially the same as the certaindivergent route on one or more values returned by counters associated todivergent routes. For example, given the certain divergent route, theroute counter 360 returns a sum of counters associated with divergentroutes that are essentially the same as the certain divergent route.

The template manipulator 362 is configured to manipulate the testscenario template according to the divergent routes in the database 358that are essentially the same as the certain divergent route, generatinga manipulated template 363. In one embodiment, the template manipulator362 manipulates the template only if the number of divergent routesstored in the database 358 that are essentially the same as the certaindivergent route reaches a predetermined threshold.

In one embodiment, the predetermined threshold is selected to have acertain value that signifies a likelihood of general usefulness of acertain divergent route; thus, the fact that the divergent routesessentially the same as the certain divergent route have a sufficientlylarge utilization (as indicated by reaching the predeterminedthreshold), indicates that the divergent routes are likely to be usefulfor other organizations too.

In one embodiment, the predetermined threshold may be a fixed value. Forexample, two divergent routes, i.e., two different runs of testscenarios need to have been identified as containing the essentially thesame route as the certain divergent route in order for the predeterminedthreshold to be reached.

In another embodiment, the predetermined threshold may be proportionalto various factors such as the number of the runs of the test scenarios,the number of users that ran the test scenarios, and/or the number oforganizations associated with the runs of the test scenarios. In oneexample, the predetermined threshold increases with the number ofdifferent organizations, thus for instance, if there are 10 differentorganizations the predetermined threshold may be 3, but if there are 100different organizations, the predetermined threshold may be 10.

In one embodiment, manipulating a test scenario template by the templatemanipulator 362 involves updating the template so the manipulatedtemplate 363 is different from the template prior to the updating. Inone example, updating the test scenario template may involve adding oneor more elements found in the divergent routes to the test scenariotemplate. In another example, updating the test scenario template mayinvolve removing one or more elements found in the test scenariotemplate that are not found in the divergent routes. In yet another,updating the test scenario template may involve changing order of one ormore elements found in the test scenario template, according to an orderof elements found in the divergent routes.

In another embodiment, manipulating a test scenario template by thetemplate manipulator 362 involves generating a new test scenariotemplate based on the test scenario template and one or more of thedivergent routes, which is considered the manipulated template 363.Thus, the manipulated template 363 may refer to an updated template or anewly generated template, depending on what manipulation the templatemanipulator 362 is configured to perform.

In one example, the test scenario template 794 illustrated in FIG. 3 isupdated according to divergent routes. In this example, template 794involves two transactions (denoted “Tx1” and “Tx2” in FIG. 3) whichinclude five screens (denoted “Scr 1” to “Scr 5” in FIG. 3). Based onruns of test scenarios by different users from different organizations,it may be determined that there is a route 796 that diverges from theroute of the template 794. According to the divergent route, many usersrun an additional screen 795 belonging to Tx1 (the screen 795 denoted by“Scr 7” in FIG. 3). Therefore, the template 794 may be manipulated tocreate a manipulated template 797 which includes the screen 795 in theappropriate place. Optionally, the manipulated template 797 replaces thetemplate 794. Alternatively, the manipulated template 797 may be createdin addition to the template 794.

In one embodiment, the computer system optionally includes a datacleaner 366 configured to select a value from the manipulated testscenario template 363, and remove the selected value from themanipulated template 363 if the selected value does not appear in runsof test scenarios that follow at least two divergent routes that areessentially the same as the certain divergent route. Optionally, thedata cleaner 366 is also configured to check that the at least twodivergent routes are associated with at least two differentorganizations. For example, there is at least a first run of a testscenario that follows a first registered divergent route that isessentially the same as the certain divergent route; there is at least asecond run of a test scenario that follow a first divergent route thatis essentially the same as the certain divergent route that follows asecond registered divergent route that is essentially the same as thecertain divergent route; and a first organization is associated with thefirst run, and a second different organization, is associated with thesecond run.

In another embodiment, the computer system optionally includes a rankingmodule 364 configured to rank templates, such as, the manipulated testscenario template 363. Optionally, ranking the manipulated template 363is done according to the number of its corresponding divergent routes.For example, the more divergent routes corresponding to a manipulatedtemplate 363, the higher the manipulated template 363 is ranked. In oneexample, ranking involves ordering templates according to number oftheir corresponding divergent routes. In another example, rankinginvolves assigning templates scores proportional to the number of theircorresponding divergent routes. In still another example, rankinginvolves ordering templates according to number of differentorganizations corresponding to the divergent routes. The larger thenumber of organizations associated with runs that had divergent routesfound to be essentially the same as a route corresponding to themanipulated template 363, the higher the rank of the manipulatedtemplate 363.

In one embodiment, the computer system optionally includes a monitoringmodule 352 that is configured to monitor the users 350 belonging to thedifferent organizations and to the activity data 353 obtained frommonitoring the users 350 to the test identifier 354.

In another embodiment, the computer system optionally includes a userinterface 368 configured to suggest to a user to run an instantiationmanipulated test scenario template. Optionally, the user interface 368may initiate the instantiation of the manipulated test scenariotemplate; for example, the user interface 368 may present a first screenbelonging to the manipulated test scenario template and prompt a user totake a certain action to advance execution.

In one embodiment, the route analyzer 356 is configured to receive a runof a test scenario this is instantiated from a first test scenariotemplate. The first test scenario template defines a first templateroute that instantiations of the first test scenario template aresupposed to follow. The route analyzer 356 is also configured todetermine whether the run follows a certain divergent route thatdiverges from the first template route. Optionally, if the it isdetermined that the run does follow the certain divergent route, thecertain divergent route is stored in the database 358.

In one embodiment, the route counter 360 is configured to receive asecond template route belonging to a second test scenario template. Forexample, there is a second run of a test scenario that is aninstantiation of the second test scenario template; the second runfollows a second template route defined by the second test scenariotemplate. Optionally, the second test scenario template is associatedwith a certain organization that does not belong to the differentorganizations; for example, a user belonging to the certain organizationran the run of the second test scenario. Optionally, the route counter360 is also configured to count number of divergent routes in thedatabase 358 that are similar to second template route.

In one embodiment, the template manipulator 362 is configured tomanipulate the second test scenario template according to the divergentroutes in the database that are similar to the second template route.Optionally, the template manipulator 362 performs the manipulation ifthe number of divergent routes reaches a predetermined threshold.Optionally, the template manipulator 362 is also configured not tomanipulate the second test scenario template according to the divergentroutes if the number does not reach the predetermined threshold.

In one example, the software systems are screen based, and similardivergent routes start from, and end with, the same screens. In anotherexample, the software systems are screen based, and similar divergentroutes have the same chain of screens but with different user keys. Instill another example, similar divergent routes have the same sequenceof screens, but possibly utilize different default values for at leastsome fields displayed on the screens.

It is to be noted that different embodiments may implement components ofthe computer systems illustrated in FIG. 1 and/or FIG. 2 in differentways. For example, in some embodiments one or more of the following maybe implemented by the same software module and/or software modulesrunning on the same processor: the monitoring module 352, the testidentifier 354, the route analyzer 356, the database 358, the routecounter 360, the template manipulator 362, the ranking module 364, thedata cleaner 366, and the user interface 368. Optionally, one or more ofthe aforementioned components may be implemented on a remote server,such as a cloud-based server.

In one example, the route analyzer 356 and/or the route counter 360 areimplemented, at least in part, as part of the database 358. For example,they may involve software modules that belong to the database 358. Inanother example, the route analyzer 356 and the route counter 360 areimplemented by the same software module that interacts with the database358. In yet another example, the ranking module 364 is implemented aspart of the template manipulator 362.

In one embodiment, the test identifier 354, the route analyzer 356, thedatabase 358, the route counter 360, and/or the template manipulator areimplemented as a cloud-based service that receives the activity data 353of the users and manipulates templates to better suit the needs of theusers. For example, the manipulated routes may test elements in a newand/or different way. Optionally, the ranking module 364 and/or the datacleaner 366 may also be part of the cloud-based service.

FIG. 4 illustrates one embodiment of a computer implemented method forutilizing divergent routes identified in runs of test scenarios tomanipulate a test scenario template. The illustrated embodiment includesthe following steps:

In step 371, identifying runs of test scenarios run by users belongingto the different organizations on software systems associated with thedifferent organizations.

In step 372, receiving a certain run of a test scenario instantiatedfrom a test scenario template; the certain run of the test scenario wasrun on a software system associated with a certain organization that isnot one of the different organizations. Optionally, the differentorganizations and the certain organization are associated with differentfields of operation. Alternatively, the different organizations and thecertain organization are associated with a same field of operation.

Optionally, the test scenario template defines a template route thatinstantiations of the template are supposed to follow. Optionally, thetemplate route includes a sequence of one or more elements involved inrunning an instantiation of the template. Optionally, an element may bea test step, a transaction, or a screen.

In step 373, identifying that the certain run follows a certaindivergent route that diverges from the template route. Optionally, adivergent route is characterized by a different sequence of elementscompared to sequence of elements defined by the template route.

In step 374, storing the certain divergent route in the database 358.

In step 375, counting number of divergent routes in the database 358that are essentially the same as the certain divergent route.

And in step 377, if the number reaches a predetermined threshold,manipulating the test scenario template according to the divergentroutes in the database that are essentially the same as the certaindivergent route. Optionally, the predetermined threshold is selectedsuch that reaching the predetermined threshold indicates that theessentially the same divergent routes are also likely to be useful forother organizations. Optionally, the predetermined threshold is at leasttwo divergent routes.

Optionally, if the number does not reach the predetermined threshold,step 377 involves refraining from manipulating the test scenariotemplate according to the divergent routes. Not reaching thepredetermined threshold indicates that the divergent routes are notlikely to be a suitable option for an organization associated with runsthat follow the certain divergent route. Additionally, a number thatdoes not reach the predetermined threshold may indicate that thedivergent routes are too specific, and are likely useful primarily forthe organizations that ran them.

In one embodiment, manipulating the test scenario template according tothe divergent routes involves updating the test scenario templateaccording to one or more of the divergent routes; for example, a newscreen belonging to the divergent routes, which did not appear in thetemplate, is added to the test scenario template.

In another embodiment, manipulating the test scenario template accordingto the divergent routes involves generating a new test scenario templatebased on the test scenario template and one or more of the divergentroutes. For example, elements from the divergent routes are combinedwith elements in the existing test scenario template in order togenerate a new template with a combination of elements that is notpossessed by the test scenario template and possibly any of thedivergent routes.

In one embodiment, identifying in the runs divergent routes in step 373involves identifying divergent routes that diverge from the templateroute and later on converge back to the template route. Additionally oralternatively, identifying in the runs divergent routes in step 373involves identifying divergent routes that diverge from the templateroute and do not converge back to the template route. Additionally oralternatively, identifying in the runs divergent routes in step 373involves identifying divergent routes that merge with the templateroute.

In one embodiment, the computer implemented method includes anadditional optional step 378 involving suggesting to a user to run aninstantiation of the updated test scenario template. For example, theuser interface 368 may utilized to present the updated template to theuser.

In one embodiment, the computer implemented method includes an optionalstep 370, which involves monitoring the users and providing dataobtained from the monitoring for use in the identifying of the runs oftest scenarios.

In another embodiment, the computer implemented method includes anoptional step 377 that involves selecting a value from the manipulatedtest scenario template, and removing the selected value from themanipulated test scenario template if the selected value does not appearin runs that follow at least two of the divergent routes. Optionally,step 377 also involves testing that the at least two of the divergentroutes are associated with at least two different organizations.

In one embodiment, the computer implemented method illustrated in FIG. 4includes an optional step of ranking the manipulated test scenariotemplate according to the number of the divergent routes. Additionallyor alternatively, ranking the manipulated test scenario template may beaccording to the number of different organizations associated with thedivergent routes.

FIG. 5 illustrates another embodiment of a computer implemented methodfor utilizing divergent routes identified in runs of test scenarios tomanipulate a test scenario template. The illustrated embodiment includesthe following steps:

In step 381, identifying runs of test scenarios run by users belongingto the different organizations on software systems associated with thedifferent organizations.

In step 382, receiving a certain run of a test scenario instantiatedfrom a first test scenario template; the first test scenario templatedefines a first template route that instantiations of the first testscenario template are supposed to follow. Optionally, the first templateroute includes a sequence of one or more elements involved in running aninstantiation of the first template. Optionally, an element may be atest step, a transaction, or a screen.

In step 383, identifying that the certain run follows a certaindivergent route that diverges from the first template route. Optionally,a divergent route is characterized by a different sequence of elementscompared to sequence of elements defined by the template route.

In step 384, storing the certain divergent route in the database 358.

In step 385, receiving a second template route belonging to a secondtest scenario template; the second test scenario template is associatedwith an organization that does not belong to the differentorganizations. Optionally, receiving the second template route involvesreceiving elements included in the second template route. Additionallyor alternatively, receiving the second template route may involvereceiving the second test scenario template that defines the secondtemplate route. Additionally or alternatively, receiving the secondtemplate route may involve receiving a run of a test scenario thatfollows the second template route, such as a run that is aninstantiation of the second template route.

In step 386, counting number of divergent routes in the database 358that are essentially the same as the second template route.

And in step 387, if the number reaches a predetermined threshold,manipulating the test scenario template according to the divergentroutes in the database that are essentially the same as the secondtemplate route. Optionally, the predetermined threshold is selected suchthat reaching the predetermined threshold indicates that the essentiallythe divergent routes are also likely to be useful for otherorganizations. Optionally, the predetermined threshold is at least twodivergent routes.

Optionally, if the number does not reach the predetermined threshold,step 387 involves refraining from manipulating the test scenariotemplate according to the divergent routes. Not reaching thepredetermined threshold indicates that the divergent routes are notlikely to be a suitable option for an organization associated with runsthat follow the certain divergent route. Additionally, a number thatdoes not reach the predetermined threshold may indicate that thedivergent routes are too specific, and are likely useful primarily forthe organizations that ran them.

In one embodiment, identifying in the runs divergent routes in step 383involves identifying divergent routes that diverge from the templateroute and later on converge back to the template route. Additionally oralternatively, identifying in the runs divergent routes in step 383involves identifying divergent routes that diverge from the templateroute and do not converge back to the template route. Additionally oralternatively, identifying in the runs divergent routes in step 383involves identifying divergent routes that merge with the templateroute.

In one embodiment, the computer implemented method includes anadditional optional step 389 involving suggesting to a user to run aninstantiation of the updated test scenario template. For example, a userinterface may utilized to present the updated template to the user.

In one embodiment, the computer implemented method includes an optionalstep 380, which involves monitoring the users and providing dataobtained from the monitoring for use in the identifying of the runs oftest scenarios.

In another embodiment, the computer implemented method includes anoptional step 388 that involves selecting a value from the manipulatedtest scenario template, and removing the selected value from themanipulated test scenario template if the selected value does not appearin runs that follow at least two of the divergent routes. Optionally,step 388 also involves testing that the at least two of the divergentroutes are associated with at least two different organizations.

In one embodiment, the computer implemented method illustrated in FIG. 5includes an optional step of ranking the manipulated test scenariotemplate according to the number of the divergent routes. Additionallyor alternatively, ranking the manipulated test scenario template may beaccording to the number of different organizations associated with thedivergent routes.

In one embodiment, a non-transitory computer-readable medium storesprogram code that may be used by a computer to utilize divergent routesidentified in runs of test scenarios to manipulate a test scenariotemplate. The computer includes a processor, and the non-transitorycomputer-readable medium stores the following program code:

Program code for identifying runs of test scenarios run by usersbelonging to the different organizations on software systems associatedwith the different organizations.

Program code for receiving a certain run of a test scenario run on asoftware system associated with a certain organization that does notbelong to the different organizations; the certain run is instantiatedfrom a test scenario template and the test scenario template defines atemplate route that instantiations of the test scenario template aresupposed to follow.

Program code for identifying that the certain run follows a certaindivergent route that diverges from the template route.

Program code for storing the certain divergent route in a database.

Program code for counting number of divergent routes in the databasethat are essentially the same as the certain divergent route.

And program code for manipulating the test scenario template accordingto the divergent routes in the database that are essentially the same asthe certain divergent route if the number reaches a predeterminedthreshold.

In one embodiment, the program code for manipulating the test scenariotemplate according to the divergent routes includes program code forupdating the test scenario template according to one or more of thedivergent routes.

In another embodiment, the program code for manipulating the testscenario template according to the divergent routes include program codefor generating a new test scenario template based on the test scenariotemplate and one or more of the divergent routes.

In one embodiment, the program code stored in the non-transitorycomputer-readable medium optionally includes program code for refrainingfrom manipulating the test scenario template according to the divergentroutes if the number does not reach the predetermined threshold. Notreaching the predetermined threshold indicates that the divergent routesare not likely to be a suitable alternative to the second templateroute, for other organizations.

In one embodiment, the program code stored in the non-transitorycomputer-readable medium optionally includes program code for selectinga value from the manipulated test scenario template, and removing theselected value from the manipulated test scenario template if theselected value does not appear in runs that follow at least two of thedivergent routes. Optionally, the program code also involves testingthat the at least two of the divergent routes are associated with atleast two different organizations.

In another embodiment, a non-transitory computer-readable medium storesprogram code that may be used by a computer to utilize divergent routesidentified in runs of test scenarios to manipulate a test scenariotemplate. The computer includes a processor, and the non-transitorycomputer-readable medium stores the following program code:

Program code for identifying runs of test scenarios run by usersbelonging to the different organizations on software systems associatedwith the different organizations.

Program code for receiving a certain run of a test scenario; the certainrun is instantiated from a first test scenario template and the firsttest scenario template defines a first template route thatinstantiations of the first test scenario template are supposed tofollow.

Program code for identifying that the certain run follows a certaindivergent route that diverges from the first template route.

Program code for storing the certain divergent route in a database.

Program code for receiving a second template route belonging to a secondtest scenario template. The second test scenario template is associatedwith an organization that does not belong to the differentorganizations.

Program code for counting number of divergent routes in the databasethat are essentially the same as the second template route.

And program code for manipulating the second test scenario templateaccording to the divergent routes in the database that are essentiallythe same as the second template route if the number reaches apredetermined threshold.

In one embodiment, the program code for manipulating the second testscenario template according to the divergent routes includes programcode for updating the second test scenario template according to one ormore of the divergent routes.

In another embodiment, the program code for manipulating the second testscenario template according to the divergent routes includes programcode for generating a new test scenario template based on the secondtest scenario template and one or more of the divergent routes.

In one embodiment, the program code stored in the non-transitorycomputer-readable medium optionally includes program code for refrainingfrom manipulating the second test scenario template according to thedivergent routes if the number does not reach the predeterminedthreshold. Not reaching the predetermined threshold indicates that thedivergent routes are not likely to be a suitable alternative to thesecond template route, for other organizations.

In one embodiment, the program code stored in the non-transitorycomputer-readable medium optionally includes program code for selectinga value from the manipulated second test scenario template, and removingthe selected value from the manipulated second test scenario template ifthe selected value does not appear in runs that follow at least two ofthe divergent routes. Optionally, the program code also involves testingthat the at least two of the divergent routes are associated with atleast two different organizations.

FIG. 6 and FIG. 7 illustrate embodiments of a computer system configuredto utilize routes followed by runs of test scenarios to manipulate atest scenario template. The illustrated embodiments include at least atest identifier 354, a route analyzer 391, a database 392, a routeretriever 393, a subset selector 394, and a template manipulator 395.

Herein, a route defines a sequence of one or more elements involved in arun of a test scenario; an element may be a test step, a transactions,or a screen. Additionally, a test scenario template may define atemplate route that instantiations of the template are supposed tofollow. Thus, a run of a test scenario based on a template is expectedto reflect the template route; for example, the run may describe screensand/or transactions included in the template route.

The test identifier 354 is configured to identify runs of test scenariosrun by users belonging to different organizations on software systemsassociated with the different organizations. Optionally, the runs areidentified from activity data 353 obtained from monitoring the users.

In one embodiment, the route analyzer 391 is configured to receive a runof a test scenario and to identify a route that the run follows; the runmay be one of the runs identified by the test identifier 354.Optionally, the route analyzer 391 identifies in the run certainelements that may include test steps executed while running the testscenario, transactions and/or businesses executed while running the testscenario, and/or screens presented to a user running the test scenario.The certain elements may also be used to describe the route. In thiscase, the run may be referred to as “following the route”.

Optionally, the route analyzer 391 may identify multiple routes that therun follows. For example, the multiple routes may contain differenttypes of elements; e.g., a first route may include screens, while asecond route may include transactions. Thus, the run be referred to asfollowing both routes. Additionally or alternatively, the route analyzer391 may include in a route a subset of the elements identified in therun. For example, the route analyzer 391 may exclude certain screensfrom a route if those screens are utilized only by one organizationand/or contain primarily proprietary data.

In one embodiment, the database 392 is configured to store the routesidentified by the route analyzer 391. Optionally, at least some of thefunctionality of the route analyzer 391 is performed by the database392; for example, part of the identification of routes is performed bysoftware and/or hardware that is part of the database (e.g., running ona server belonging to the database 392), and/or is controlled by thedatabase (e.g., a module of the database 392 may control some of theprocesses performed by the route analyzer 391).

In one example, each time a route is identified in a run of a testscenario, the route is stored in the database 392. Optionally, storing aroute involves storing elements included in the route, a run of a testscenario that follows the route, information pertaining to a user thatran a test scenario that followed the route, and/or informationpertaining to an organization associated with a run of a test scenariothat followed the divergent route.

In another example, each time a route is identified in a run of a testscenario, the route analyzer 391 and/or the database 392 determinewhether another route essentially the same as the route has been alreadystored. If no essentially same route has already been stored, the routeis stored in the database 392.

Optionally, if another essentially same route has already been stored,then a certain counter associated to the already stored route isincremented. Optionally, such a counter may be a counter correspondingto runs following the stored route, a counter corresponding to usersthat ran runs following the stored route, and/or a counter correspondingto organizations associated with runs following the stored route.

The route retriever 393 is configured to receive a certain templateroute 390 belonging to a test scenario template and to retrieve from thedatabase 392 divergent routes that diverge from the template route 390.The test scenario template defines the template route 190, which is aroute that instantiations of the template are supposed to follow. Thatis, each run of a test scenario instantiated from the template issupposed to reflect a certain sequence of elements, as defined by theroute. Optionally, the route retriever 393 receives the test scenariotemplate to which the route 390 belongs, and derives the route 390 fromthe template. Additionally and or alternatively, the route retriever 393may receive a representation of the template route 390.

In one embodiment, at least some of the functionality of the routeretriever 393 is performed by the database 392; for example, part of theidentification of divergent routes is performed by software and/orhardware that is part of the database (e.g., running on a serverbelonging to the database 392), and/or is controlled by the database(e.g., a module of the database 392 may control some of the processesperformed by the route retriever 393).

In one embodiment, the test scenario template received by the routeretriever 393 is associated with a certain organization that does notbelong to the different organizations. Optionally, a test scenariotemplate may be considered associated with an organization if a userbelonging to the organization ran an instantiation of the templateand/or is intended to run an instantiation of the template. Additionallyor alternatively, a test scenario template may be considered associatedwith an organization if the template was generated based on one or moreruns of test scenarios that are associated with the organization.Optionally, the different organizations and the certain organization areassociated with different fields of operation (e.g., the differentorganizations are in the insurance field, while the certain organizationis in the field of automobile manufacture). Alternatively, the differentorganizations and the certain organization are associated with a samefield of operation.

Based on the elements included in the template route 390, the routeretriever 393 may identify in the database 392 routes that diverge fromthe template route 390, and as such are considered divergent routes withrespect to the template route 390. Optionally, a divergent route thatdiverges from a template route 390 is characterized by a differentsequence of elements compared to sequence of elements defined by thetemplate route 390. For example, the route retriever 393 may compareelements in the template route 390 with elements belonging to runs inthe database 392 in order to determine whether some elements beingcompared are similar and/or appear in the same order as the elements inthe template route 390.

In one embodiment, the route retriever 393 is configured to identifydivergent routes that diverge from the template route 390 and later onconverge back to the template route 390. Additionally or alternatively,the route retriever 393 may also be configured to identify divergentroutes that diverge from the template route 390 and do not converge backto the template route 390. Additionally or alternatively, the routeretriever 393 may be also configured to identify divergent routes thatmerge with the template route 390.

In one embodiment, an essentially exhaustive search is performed in thedatabase to identify the divergent routes. Optionally, essentially allroutes stored in the database 392 are evaluated and/or considered whensearching for the divergent routes.

In another embodiment, searching for the divergent routes in thedatabase 392 involves evaluating and/or considering a portion of theroutes stored in the database (e.g., randomly selecting 10% of theroutes in the database 392 and examining them to identify the divergentroutes). Optionally, at least some routes belonging to the portion ofthe routes represent other routes (e.g., they are representatives ofgroups of related routes). Thus, if a representative route is deemed adivergent route, or close to being a divergent route, other routes inits group may be evaluated too.

In another embodiment, searching for the divergent routes in thedatabase 392 involves filtering routes in the database according toelements in the template route 390. For example, the database 392 may bequeried for routes that include a certain element belonging to thetemplate route 390, and only routes containing the certain element areevaluated.

The subset selector 394 is configured to select from the divergentroutes retrieved by the route retriever 393 a subset of divergent routesthat are similar to each other. Optionally, the routes belonging to thesubset are essentially the same. Optionally, the subset includes all thedivergent routes retrieved by the route retriever 393. Alternatively,the subset includes at least one of the divergent routes, and at leastone divergent route is not included in the subset. Optionally, thesubset includes exactly one divergent route; alternatively, the subsetmay include two or more divergent routes.

In one embodiment, the subset includes divergent routes that are similarto each other, according to some measure of similarity. Optionally, thedivergent routes in the subset are less similar to other divergentroutes not in the subset, according the measure of similarity. In oneexample, the software systems are screen based, and similar divergentroutes start from, and end with, the same screens. In another example,the software systems are screen based, and similar divergent routes havethe same chain of screens but with different user keys. In still anotherexample, similar divergent routes have the same changes in relation tothe template route from which they arrived.

In one example, divergent routes belonging to the subset have a certainnumber of elements in common. For example, all divergent routes in thesubset share a number of common screens. In another example, divergentroutes belonging to the subset all have a minimal pairwise similaritybetween pairs of divergent routes in the subset. For example, the anglebetween a vector representation of any two divergent routes in thesubset is smaller than a certain threshold. In yet another example,divergent routes returned by the route retriever 393 are clustered andthe subset selector 394 selects a certain cluster of divergent routes asthe subset.

In one embodiment, the subset selector 394 compares the divergent routesretrieved by the route retriever 393 to the template route 390, andincludes in the subset only divergent routes that show at least acertain level of similarity to the template route 390. For example, alldivergent routes must share a certain number of elements with thetemplate route 390 (in addition to being similar to each other).

The template manipulator 395 is configured to manipulate the testscenario template according to the subset of divergent routes if thesize of the subset reaches a predetermined threshold. A result of themanipulation may be a manipulated template 399. Optionally, the templatemanipulator 395 is also configured not to manipulate the test scenariotemplate according to subset if the size of the subset does not reachthe predetermined threshold.

In one embodiment, the predetermined threshold is selected to have acertain value that signifies a likelihood of general usefulness ofdivergent routes belonging to the subset; thus, the fact that thedivergent routes in the subset have a sufficiently large utilization (asindicated by the size of the subset), indicates that the divergentroutes in the subset are likely to be useful for other organizationstoo. If the subset is too small, this may indicate that the routes inthe subset are utilized by a small number of users and/or organizations,and thus are not likely useful for other organizations.

In one embodiment, the predetermined threshold may be a fixed value. Forexample, a subset of at least two divergent routes. In anotherembodiment, the predetermined threshold may be proportional to variousfactors such as the number of the runs of the test scenarios, the numberof users that ran the test scenarios, and/or the number of organizationsassociated with the runs of the test scenarios. In one example, thepredetermined threshold increases with the number of differentorganizations, thus for instance, if there are 10 differentorganizations the predetermined threshold may be a subset of at leastsize 3, but if there are 100 different organizations, the predeterminedthreshold may be a subset of at least 10 divergent routes.

In one embodiment, manipulating a test scenario template by the templatemanipulator 395 involves updating the template so the manipulatedtemplate 399 is different from the template prior to the updating. Inone example, updating the test scenario template may involve adding oneor more elements found in the divergent routes in the subset to the testscenario template. Optionally, the one or more elements that are addedare part of at least a predetermined proportion of the divergent routesbelonging to the subset (e.g., the added elements appear in at least 50%of the divergent routes in the subset). Optionally, the one or moreelements that are added are part of all the divergent routes belongingto the subset. In another example, updating the test scenario templatemay involve removing one or more elements found in the test scenariotemplate that are not found in the divergent routes in the subset.Optionally, the one or more elements that are removed are not part of atleast a predetermined proportion of the divergent routes belonging tothe subset (e.g., the added elements appear in less than 50% of thedivergent routes in the subset). Optionally, the one or more elementsthat are remove do not appear in any of the divergent routes belongingto the subset. In yet another, updating the test scenario template mayinvolve changing order of one or more elements found in the testscenario template, according to an order of elements found in thedivergent routes in the subset.

In another embodiment, manipulating a test scenario template by thetemplate manipulator 395 involves generating a new test scenariotemplate based on the test scenario template and one or more of thedivergent routes in the subset. For example, the new template mayinclude at least some of the elements from the test scenario templateand at least some elements from divergent routes belonging to thesubset. Optionally, the new template is generated by copying a pervioustemplate and updating the copy.

In one embodiment, the computer system may optionally include a datacleaner 397 configured to select a value from the manipulated testscenario template 399, and remove the selected value from themanipulated template 399 if the selected value does not appear in runsof test scenarios that follow at least two divergent routes that areessentially the same as the certain divergent route. Optionally, thedata cleaner 397 is also configured to check that the at least twodivergent routes are associated with at least two differentorganizations. For example, there is at least a first run of a testscenario that follows a first registered divergent route that isessentially the same as the certain divergent route; there is at least asecond run of a test scenario that follow a first divergent route thatis essentially the same as the certain divergent route that follows asecond registered divergent route that is essentially the same as thecertain divergent route. Additionally, a first organization isassociated with the first run, and a second different organization, isassociated with the second run. Optionally, the manipulated template 399may refer to an updated template or a newly generated template,depending on what manipulation the template manipulator 395 isconfigured to perform.

In one embodiment, the computer system may also optionally include amonitoring module 352 that is configured to monitor the users 350belonging to the different organizations and to provide the activitydata 353 obtained from monitoring the users 350 to the test identifier354.

In another embodiment, the computer system may also optionally include auser interface 398 configured to suggest to a user to run aninstantiation manipulated test scenario template. Optionally, the userinterface 398 may initiate the instantiation of the manipulated testscenario template; for example, the user interface 398 may present afirst screen belonging to the manipulated test scenario template andprompt a user to take a certain action to advance execution.

In another embodiment, the computer system may also optionally include aranking module 396 configured to rank templates, such as, themanipulated test scenario template 399. Optionally, ranking themanipulated template 399 is done according to the size of the subset ofdivergent routes selected by the subset selector 394; for example, thelarger the subset, the higher the manipulated template 399 is ranked. Inone example, ranking involves ordering templates according to size oftheir corresponding subset of divergent routes. In another example,ranking involves assigning templates scores proportional to the size oftheir corresponding subset of divergent routes. In still anotherexample, ranking involves ordering templates according to number ofdifferent organizations associated with routes belonging to theircorresponding subsets. The larger the number of organizations associatedwith runs that followed divergent routes in a subset, the higher therank of the manipulated template 399.

It is to be noted that different embodiments may implement components ofthe computer systems illustrated in FIG. 6 and/or FIG. 7 in differentways. For example, in some embodiments one or more of the following maybe implemented by the same software module and/or software modulesrunning on the same processor: the monitoring module 352, the testidentifier 354, the route analyzer 391, the database 392, the routeretriever 393, the subset selector 394, the template manipulator 395,the ranking module 396, the data cleaner 397, and the user interface398. Optionally, one or more of the aforementioned components may beimplemented on a remote server, such as a cloud-based server.

In one example, the route analyzer 391, the route retriever 393, and/orthe subset selector 394 may be implemented, at least in part, as part ofthe database 392. For example, they may involve software modules thatbelong to the database 392. In another example, the route analyzer 391,the route retriever 393, and/or the subset selector 394 may beimplemented by the same software module that interacts with the database392. In yet another example, the ranking module 396 and/or the datacleaner 397 are implemented as part of the template manipulator 395.

In one embodiment, the test identifier 354, the route analyzer 391, thedatabase 392, the route retriever 393, the subset selector 394, and/orthe template manipulator 395, are implemented as a cloud-based servicethat receives the activity data 353 of the users and manipulatestemplates to better suit the needs of the users. For example, themanipulated routes may test elements in a new and/or different way.Optionally, the ranking module 396 and/or the data cleaner 397 may alsobe part of the cloud-based service.

FIG. 8 illustrates one embodiment of a computer implemented method forutilizing routes followed by runs of test scenarios to manipulate a testscenario template. The illustrated embodiment includes the followingsteps:

In step 401, identifying runs of test scenarios run by users belongingto different organizations on software systems associated with thedifferent organizations.

In step 402, receiving a certain run of a test scenario; the certain runmay be one of the runs identified in step 401.

In step 403, identifying a route that the certain run follows.Optionally, a route defines a sequence of one or more elements involvedin running a test scenario. Optionally, an element is selected from thegroup consisting of test steps, transactions, and screens.

In step 404, storing route in the database 392.

In step 405, receiving a certain template route belonging to a testscenario template; the test scenario template is associated with acertain organization that does not belong to the differentorganizations. Optionally, the different organizations and the certainorganization are associated with different fields of operation. Forexample, the certain organization is in the travel business, while thedifferent organizations are in finances, health services, or automobilemanufacturing. Alternatively, the different organizations and thecertain organization are associated with a same field of operation. Forexample, they all operate in the field of internet commerce.

In step 406, retrieving from the database 392 divergent routes thatdiverge from the template route. Optionally, a divergent route ischaracterized by a different sequence of elements compared to sequenceof elements defined by the template route. Optionally, the divergentroutes involve identifying routes that diverge from the template routeand later on converge back to the template route. Additionally oralternatively, retrieving the divergent routes may involve identifyingroutes that diverge from the template route and do not converge back tothe template route. Additionally or alternatively, retrieving thedivergent routes may involve identifying divergent routes that mergewith the template route.

In step 407, selecting from the divergent routes a subset of divergentroutes that are similar to each other.

And in step 408, if the size of the subset reaches a predeterminedthreshold, manipulating the test scenario template according to thesubset. Optionally, the predetermined threshold is selected such thatreaching the predetermined threshold indicates that the essentially thedivergent routes belonging to the subset are also likely to be usefulfor other organizations. Optionally, the predetermined threshold is atleast two divergent routes.

Optionally, if the size of the subset does not reach the predeterminedthreshold, step 408 involves refraining from manipulating the testscenario template according to the subset. Not reaching thepredetermined threshold indicates that the divergent routes are notlikely to be a suitable option for an organization associated with runsthat follow the certain divergent route. Additionally, a number thatdoes not reach the predetermined threshold may indicate that thedivergent routes are too specific, and are likely useful primarily forthe organizations that ran them.

In one embodiment, manipulating the test scenario template according tothe subset involves updating the test scenario template according to oneor more of the divergent routes in the subset; for example, a new screenbelonging to the divergent routes in the subset, which did not appear inthe template, is added to the test scenario template.

In another embodiment, manipulating the test scenario template accordingto the subset involves generating a new test scenario template based onthe test scenario template and one or more of the divergent routesbelonging to the subset. For example, elements from the divergent routesin the subset are combined with elements in the existing test scenariotemplate in order to generate a new template with a combination ofelements that is not possessed by the test scenario template andpossibly any of the divergent routes in the subset.

In one embodiment, the computer implemented method includes anadditional optional step 410 involving suggesting to a user to run aninstantiation of the updated test scenario template. For example, theuser interface 398 may utilized to present the updated template to theuser.

In one embodiment, the computer implemented method includes an optionalstep 400, which involves monitoring the users and providing the activitydata 353 obtained from the monitoring for use in the identifying of theruns of test scenarios.

In another embodiment, the computer implemented method includes anoptional step 409 that involves selecting a value from the manipulatedtest scenario template, and removing the selected value from themanipulated test scenario template if the selected value does not appearin runs that follow at least two of the divergent routes belonging tothe subset. Optionally, step 409 also involves testing that the at leasttwo of the divergent routes are associated with at least two differentorganizations.

In one embodiment, the computer implemented method illustrated in FIG. 8includes an optional step of ranking the manipulated test scenariotemplate according to the number of the divergent routes belonging tothe subset. Additionally or alternatively, ranking the manipulated testscenario template may be done according to the number of differentorganizations associated with the divergent routes belonging to thesubset.

In one embodiment, a non-transitory computer-readable medium storesprogram code that may be used by a computer to utilize routes followedby runs of test scenarios to manipulate a test scenario template. Thecomputer includes a processor, and the non-transitory computer-readablemedium stores the following program code:

Program code for identifying runs of test scenarios run by usersbelonging to different organizations on software systems associated withthe different organizations.

Program code for receiving a certain run of a test scenario.

Program code for identifying a route that the run follows.

Program code for storing the route in a database.

Program code for receiving a certain template route belonging to a testscenario template. Optionally, the test scenario template is associatedwith a certain organization that does not belong to the differentorganizations.

Program code for retrieving from the database divergent routes thatdiverge from the template route.

Program code for selecting from the divergent routes a subset ofdivergent routes that are similar to each other.

And program code for manipulating the test scenario template accordingto the subset if the size of the subset reaches a predeterminedthreshold.

In one embodiment, the program code for manipulating the test scenariotemplate according to the subset includes program code for updating thetest scenario template according to one or more of the divergent routesbelonging to the subset.

In another embodiment, the program code for manipulating the testscenario template according to the subset includes program code forgenerating a new test scenario template based on the test scenariotemplate and one or more of the divergent routes belonging to thesubset.

In one embodiment, the program code stored in the non-transitorycomputer-readable medium optionally includes program code for refrainingfrom manipulating the test scenario template according to the subset ifthe size of the subset does not reach the predetermined threshold. Notreaching the predetermined threshold indicates that the divergent routesbelonging to the subset are not likely to be a suitable alternative tothe template route, for other organizations.

In one embodiment, the program code stored in the non-transitorycomputer-readable medium optionally includes program code for selectinga value from the manipulated test scenario template, and removing theselected value from the manipulated test scenario template if theselected value does not appear in runs that follow at least two of thedivergent routes belonging to the subset. Optionally, the program codealso involves testing that the at least two of the divergent routes areassociated with at least two different organizations.

Monitoring Users and Identifying Runs

Some of the disclosed embodiments involve software systems that may becharacterized as being data-oriented large-scale software systems.Examples of data-oriented large-scale software systems includeEnterprise Resource Planning (ERP), such as from the followingtrademarks: SAP, Oracle Application, The Sage Group, Microsoft Dynamics,and SSA Global Technologies; billing systems (also known as revenuemanagement) such as from the following trademarks: Amdocs, ComverseInc., and Convergys Corporation; service management systems; andportfolio management systems. Installing, upgrading, and/or maintainingsuch systems require running many tests in order to validate thesystems' behavior.

In this disclosure, users are often referred to as being monitored. Thismonitoring typically concerns activity of the user on a software systembeing tested. A monitored user may be a human (e.g., performing acertain test) and/or a system module (e.g., a module initiating therunning of a certain procedure).

In one embodiment, a test scenario refers to a functional and/or atechnical specification for testing a business process and/or one ormore transactions. A test scenario may specify how to test one or morebusiness processes, business requirements, test conditions, logicaldata, and/or expected results of tests.

In one embodiment, runs of test scenarios are identified and/or obtainedbased on data collected from monitoring users. For example, monitoringof a user may involve collection of data related to inputs provided by auser to a system, along with the transactions that were run, and resultsof the transactions. This data may be used to identify runs of testscenarios that describe test steps taken by a user and a result ofexecuting the test steps on the software system.

In another embodiment, monitoring a user is done in order to obtainactivity data of a user on a software system. The activity data mayinclude data related to inputs provided by the user and/or other sourcesto the software system, outputs generated by the software system, and/orintermediate values generated by the software system (e.g., temporaryfiles generated by the system, network traffic of the system, and/orcontent of memory belonging to the system). Optionally, the activitydata may be utilized in order to identify runs of test scenarios.

In some embodiments, users perform at least part of their interactionwith a software system via a user interface that includes a display thatdisplays screens. Optionally, a screen may refer to a presentation of acertain form through which a user may access, modify and/or enter data.Optionally, a screen includes one or more fields. Optionally, a fieldmay have a certain name and/or identifier. Optionally, a field may havean associated value, which may be referred to herein as a “field value”.The field value may or may not be presented on the screen. Optionally,the user is able to edit certain field values, while other field valuesmay not be editable by the user. Optionally, certain field values may berequired values, which the user must enter before completing a screen.Optionally, field values from a screen may correspond to one or moredatabase entries. For example, a screen displaying customer details(e.g., name, address, telephone number) may correspond to a record ofthe customer in a customer database.

In one embodiment, monitoring users involves monitoring certaintransactions and/or business processes that were executed by the users,as part of running test scenarios. Optionally, monitoring users mayinvolve monitoring which programs executed by the users, along withinvocation values of the users and/or return values of the programs.

In one embodiment, monitoring a user may involve recording and/orprocessing inputs the user provides to the software system (e.g., via akeyboard, mouse click, visual cue). Additionally or alternatively, theinputs may include digital data transmitted to the system (e.g., a fileof digital data). In another example, monitoring a user may involverecording and/or processing outputs generated by the software system.Such outputs may include, values presented on a screen, written in afile, printed, and/or provided in the form of auditory cues. The outputsmay be presented to the user and/or provided to other recipients. In yetanother example, monitoring a user may involve utilization of internalstate data of the software system; data that may not have been directlyprovided by the user and may also not be directly provided to the user(e.g., memory content, database activities, and/or network traffic).

In some embodiments, users performing tests may be given instructionsand/or scripts describing some of the steps involved in a certain test(e.g., which procedures to run, what buttons to push, and/or what valuesto provide the system). Optionally, such instructions and/or scripts maybe given in electronic form (e.g., a computer file or instructions on ascreen) and may be recorded and/or processed as part of monitoring auser.

As described in some embodiments in this disclosure, monitoring a userrunning test scenarios on a software system may be performed by amonitoring module. Optionally, the monitoring module is, and/orutilizes, a software module that interacts with the software system onwhich the test scenarios are run, in order to obtain data related toactivity of the user on the software system. Optionally, the monitoringmodule is implemented, at least in part, as part of the software system.For example, the monitoring module may include one or more programs thatare part of a distribution of the software system. Additionally oralternatively, the monitoring module may be implemented, at least inpart, separately from the software system. For example, the monitoringmodule may include programs that are not part of the software system(e.g., not included in a distribution of the software system). Inanother example, the monitoring module may include programs that run onhardware that does not run the programs belonging to the softwaresystem; e.g., the monitoring module may run programs on differentservers than servers on which the software system runs programs.Optionally, the monitoring module is implemented, at least in part, onhardware remote from hardware that runs the software system. Forexample, a software system may run on hardware located at a certainorganization, while the monitoring module may run, at least in part, onremote cloud-based servers that are not associated with the certainorganization.

In some embodiments, the monitoring module may receive data from thesoftware system that is intended for the monitoring module (e.g., thesoftware system may explicitly send the monitoring module datadescribing activity on the software system). Additionally oralternatively, the monitoring module may intercept and/or examine dataof the software system that is not intended primarily for the monitoringmodule. For example, the monitoring module may sniff network trafficthat involves message exchange between modules of the software system aspart of the normal operation of the software system.

In some embodiments, a run of a test scenario may include identifiers ofbusiness processes, transactions, and/or system commands that wereexecuted while running the test scenario. Additionally or alternatively,a run of a test scenario may include values that were provided to asoftware system (e.g., input values in a screen), values generated bythe software system (e.g., outputs from a transaction), and/or internalvalues of the software system (e.g., intermediate values generated by atransaction). Additionally or alternatively, a run of a test scenariomay include state information about systems involved in running the testscenario (e.g., the state of certain system resources, and/orperformance data such as CPU load or network congestion), and/orinformation about a user running the test scenario (e.g., organizationaffiliation, department, job title, permissions). Optionally, a certaintest step, transaction, command or procedure is said to be describedand/or included in a run of a test scenario if the run of the testscenario indicates that the certain test step, transaction, command, orprocedure was executed as part of running the test scenario. Optionally,examining the run of the test scenario may reveal a value associatedwith the certain test step, transaction, command, or procedure.Additionally, examining the run of the test scenario may reveal abehavior of the system with respect to the certain test step,transaction, command, or procedure. For example, a run of a testscenario may indicate whether or not a certain transaction, involved inthe running of the test scenario, produced an error message.

In one embodiment, identifying a run of a test scenario from dataobtained from monitoring one or more users is done as part of theprocess of monitoring the users. For example, a user may be monitoredfor a session, which is a certain period of time that corresponds torunning of a certain test scenario. Each session may start when therunning of the certain test scenario starts (e.g., when a first screenis presented to a user), and may end when the running ends (e.g., afterthe user enters a last value involved in the test scenario). Thus, anydata recorded during the session may be easily identified as belongingto a run of the certain test scenario.

In one embodiment, the action of monitoring a user and the action ofidentifying a run of a test scenario from data obtained from themonitoring are performed separately. For example, monitoring maygenerate bulk data corresponding to activity of one or more users.Identifying runs in the bulk data may require parsing the data toidentify certain portions of interest, such as transactions executed byeach of the users and/or inputs and outputs of each of the users. In oneexample, identifying runs involves detecting in the data obtained frommonitoring signals that denote a start and/or end of a run. Optionally,the signals may be explicit, such as explicit tags in the data thatdenote a start of running a test and/or end of running a test scenario.Alternatively or additionally, the signals may be implicit. For example,a user entering data after not doing so for a while may signal a startof a run; while having a user stop entering data, and not enteringfurther data for a prolonged period may signal an end of a run. Inanother example, the fact that a user performs a certain transaction maysignal a start of a run, such as entering data in a screen labeled“begin new client record”. Similarly, certain transactions may signal anend of a run, such as entering data in a screen labeled “finalize newclient”. In yet another example, identifying a run of a test scenariomay involve selecting certain types of data and/or amounts of data thatare to be included in the run. For example, while monitoring a user mayinvolve collection of various types of data, in a certain system, runsof test scenarios may include only descriptions of screens presented toa user while interacting with the certain system. Thus, in this example,identifying runs may involve identifying and retaining descriptions ofscreens.

The process of identifying runs of test scenarios of a certain user mayutilize, in some embodiments, data collected from other users. Forexample, determining boundaries of a test scenario, such as determiningat what screen a run of the test scenario typically starts, and at whatscreen it ends, may be based on examination of data obtained frommultiple users. In the examination of the data, certain common screensthat appear in data from multiple users may be detected, and runs oftest scenarios may be restricted to begin and/or end with a descriptionof at least one of the common screens. Alternatively or additionally,identifying runs of test scenario run by a certain user, may involveexamination of data obtained primarily from monitoring the certain user.

As described in some embodiments in this disclosure, identifying runs oftest scenarios run by users on software systems may be performed by atest identifier. In some embodiments, the test identifier receives datafrom the monitoring module. Optionally, the test identifier andmonitoring module are realized by the same software module and/or samehardware. In one example, one program may perform at least some of theoperations involved in monitoring the users and identifying the runs. Inanother example, programs that perform at least some of the operationsinvolved in monitoring the users and identifying the runs of testscenarios, run on the same servers.

In one embodiment, the monitoring module runs, at least in part, onhardware that is different from hardware on which at least some of theactions of the test identifier are performed. For example, a monitoringmodule that monitors users belonging to a certain organization may run,at least in part, on servers belonging to the certain organization.However, a test identifier, which receives data from the monitoringmodule, may run on remote servers belonging to, or controlled by, adifferent organization, such as an organization that provides analysisservices to the certain organization.

Crowd Testing Data

Some aspects of this disclosure involve utilization of runs of testscenarios run by users belonging to, or associated with, differentorganizations. Optionally, a group of users running test scenarios maybe referred to as a “crowd” or “crowd users”. Optionally, crowd users donot run test scenarios in a coordinated effort, such as users belongingto different organizations that do cooperate to run test scenarios totest a certain software system that belongs to a certain organization,or users making a coordinated effort to test certain aspects of theirsoftware systems. Rather, as typically used in this disclosure, theusers belonging to the different organizations run test scenarios totest software systems of their respective organizations, and do soessentially independently of each other.

In some embodiments, a first organization and second organization areconsidered different organizations if the first organization has adifferent ownership structure (e.g., a different composition ofshareholders) than the second organization. Additionally oralternatively, two organizations may be considered differentorganizations if they have different employees, and/or different membersof management.

In one embodiment, a user may be considered to belong to an organizationand/or may be considered to be associated with the organization, if theuser has a certain relationship with the organization such as being anemployee of the organization, a member of the organization, and/orcontracted to work for the organization. Additionally or alternatively,a user may belong to an organization, and/or be associated with theorganization, if the worker performs work that is directly or indirectlydone on behalf of the organization. For example, an employee of a firstorganization may be considered associated with a second organization ifthe employee runs, as part of her duties, tests that involve a system ofthe second organization. As used herein, the terms “associated” and“belongs” may be used interchangeably when referring to a user beingassociated with an organization or a user belonging to an organization.

Some aspects of this disclosure involve utilization of runs of testscenarios run on software systems belonging to different organizationsand/or associated with the different organization. Optionally, asoftware system may be considered to belong to a certain organizationand/or may be considered to be associated with a certain organization ifthe software system runs, at least in part, on hardware belonging to theorganization and/or paid for by the organization (e.g., software runningon cloud-based servers billed to the organization). Additionally oralternatively, a software system may be considered to belong to anorganization and/or be associated with the organization if a user fromthe organization runs test scenarios, which run at least in part, on thesoftware system. As used herein, the terms “associated” and “belongs”may be used interchangeably when referring to a software system beingassociated with an organization or a software system belonging to anorganization.

Some embodiments in this disclosure involve monitoring multiple usersbelonging to different organizations. In some embodiments, each userbelongs to a single organization. For example, 100 different users maybelong to 30 different organizations; some organizations may have asingle user that belongs to them, while other organizations havemultiple users that belong to them. In some embodiments, users maybelong to more than one organization. For example, 100 different usersmay belong to 150 different organizations; in this example, some of theusers belong to more than one different organization.

In the embodiments, some of the systems on which test scenarios are runmay be data-oriented large-scale software systems. Optionally, thesoftware systems are packaged applications having multiple modules.Optionally the software systems may be considered similar to each otherif the software systems use in a similar manner one or more modules ofthe same type (possibly having different versions). Additionally oralternatively, the software systems may be considered similar if theycontain one or more modules with a similar function (e.g., modules thatmanage sales, stocking, or human resources).

In some embodiments, analyzing runs of test scenarios obtained frommonitoring running of test scenarios by a crowd of users, on softwaresystems of multiple organizations, may help gain certain insights thatmay not be easily attained from analysis obtained from runs associatedwith a single organization. For example, runs of test scenarios obtainedfrom monitoring a crowd may help gain insight into the generalapplicability of certain test scenarios and/or test steps for testingcertain components of the systems, such as modules, business process,and/or transactions. Based on the fact that multiple organizations runtest scenarios to test certain components, it may be postulated that adifferent organization should also run similar test scenarios; this maybe especially true if the different organization is in the same field ofoperations as the multiple organizations.

In some embodiments, analyzing runs of test scenarios obtained frommonitoring running of test scenarios on systems of multipleorganizations may also help gain insight into the commonality of certainaspects of the systems, as indicated by their reoccurrence in the runs.Such insight may be used, in some embodiments, to deduce whether acertain aspect is particular to a single organization, or a small numberof organizations from the multiple organizations. Alternatively, it maybe deduced, the certain aspect may be rather general and may beassociated with a relatively large portion of the multipleorganizations. For example, runs of test scenarios associated withmultiple organizations may include a field which is given a value ineach run of the test scenarios. If a certain value is entered in a largeproportion of the runs (the same certain value is entered in each run ofthe large proportion or runs), then the value may be considered adefault value or a general value. In this case, knowing the certainvalue may not assist much to identify a specific organization which isassociated with a run from which the certain value was taken. However,if a certain value appears only in runs of a single organization, or ina small proportion of the runs, then the value may be considered aunique and/or proprietary value. In this case, knowing the certain valuemay assist to identify a specific organization which is associated witha run from which the certain value was taken. Thus, in some embodiments,monitoring runs of test scenarios associated with multiple organizationsmay help discover non organization-specific default values that can beused for running test scenarios of other organizations. Additionally,the same analysis may help identify what data in the runs may beconsidered proprietary.

Routes and Divergent Routes

A route, as used in some embodiments described in this disclosure,defines a sequence of one or more elements involved in the running of atest scenario. Optionally, an element may be a test step, a transaction,and/or a screen. Optionally, a run of the test scenario, such as a runobtained from monitoring a user running the test scenario, describes thesequence of one or more elements involved in the running of the testscenario. For example, the run of a test scenario may describe the teststeps performed while running the test scenario, the transactionsexecuted while running the test scenario, and/or the screens presentedto a user while the user ran the test scenario.

In one embodiment, a test scenario template defines certain elementsthat are to be executed when running an instantiation of the template,and an order of execution of the elements in the instantiation. Thetemplate may include logic that determines the order according toruntime parameters. Thus, a test scenario template defines one or moreroutes (also referred to as “the template route/s”) that may beassociated with the template. Execution of an instantiation of thetemplate, i.e., running a test scenario based on the template, isexpected to follow the template route/s. For example, if a templateincludes screens 1-4 (and in that order), an instantiation of thetemplate is expected to present screen 1, followed by screen 2, screen 3and then screen 4.

In some embodiments, by examining a run of the test scenario it may bedetermined whether during the running of the test scenario a certainroute was followed or not. For example, if a sequence of one or moreelements described in the run corresponds to the sequence of the route,then it may be assumed that the run follows the route. Conversely, ifthe sequence of one or more elements described in the run diverges fromthe sequence of the route, then it may be assumed that the run divergesfrom the route.

In one embodiment, a test scenario template may have more than one routethat may be treated as a template route. Optionally, the template mayinclude alternative test steps that may depend on a choice made by auser. For example, when a user attempts to update a record, according tothe template, certain test steps are conducted if the update issuccessful, while other steps are to be conducted if the update attemptreturns an error. In another example, a screen belonging to the templateprovides the user with an option, such as setting the price of a part.Based on the price range, certain actions may be required: with a priceof below $1000 the user may continue as usual, however, a price of $1000or more requires the user to fill out a screen that generates a requestfor manager approval. Thus, when examining routes a template testscenario may define, it is possible for there to be more than onetemplate route.

There may be various relationships between routes (e.g., differentroutes identified in multiple runs of test scenarios). For example,routes may be distinct of each other (e.g., they do not contain anyessentially similar elements). In another example, a first route may becontained or partially contained in a second route (e.g., there are oneor more elements that are essentially the same in both of the first andthe second routes). In the latter case, the first and/or the secondroutes may be considered divergent routes, and/or one of the two routesmay be considered to diverge from the other. In particular, herein wemay consider a route that diverges from a template route. Optionally, incases where a certain test scenario template may have a multipletemplate routes, a route may be considered a divergent route if itdiverges from all of the multiple template routes.

A divergent route does not completely follow a route to which it iscompared, such as a template route. By “not completely follows” it ismeant that there is a substantial difference between the divergent routeand the route to which it is being compared, such as an element notshared by both routes. In one embodiment, a divergent route ischaracterized by a different sequence of elements compared to a sequenceof elements defined by the template route, and/or differentconfigurations of the transactions used by the process under test. Forexample, the sequence of the divergent route may include a transactionnot included in the sequence of the template route. In another example,the sequence of the divergent route may be missing a transaction that isin the sequence of the template route. In yet another example, thesequence of the divergent route includes at least one transaction thatis executed out of order, with respect to an order of executingtransactions that is defined by the template route.

In some embodiments, for a first route to be considered a divergentroute with respect to a second route, the first and second routes musthave a certain degree of similarity. Optionally, the first and secondroutes need to be considered similar to each other. Additionally, thefirst and second route may need to be considered essentially the same.Additional information regarding how to interpret routes that aresimilar and/or essentially the same is given below.

Routes followed by runs of test scenarios may diverge from theirrespective template routes for various reasons.

In one example, different organizations may configure and/or use theirsystems in different ways. The different configurations and/or ways ofusing the systems may change the behavior of the transactions, and as aresult, may change the test scenarios that correspond to thetransactions that behave differently. Optionally, the differentconfigurations may lead to a different choice of transactions that areto be executed in a test scenario, and/or a different order of executionof the transactions. In one example, a different choice of transactionsand/or order of execution may represent an alternative way to perform acertain task. Though a task is typically performed in a firstorganization in a first method, which corresponds to a first route, itmay be performed in another way, such as the way it is performed by asecond organizations, represented by a second route. Thus, the secondroute, which may be considered a divergent route with respect to thefirst, may be a legitimate method for performing the task, and follow aroute that a user of the first organization may actually end upfollowing (e.g., through inadvertent configuration change orserendipitous discovery of the user). Thus, it may be advantageous forthe first organization to also test the second route.

In one example, a manual test scenario is interpreted by a user andexecuted by a processor. The manual test scenario may be based on acertain template (e.g., a script given to the user), and as such,executing the script should lead to a run of the test scenario thatfollows the template route. However, as the user runs the test scenario,the user may perform certain actions that do not follow the script; forexample, the user may determine that certain test steps in the scriptare incorrect or irrelevant and thus should not be performed. Thus,decisions, reactions, and/or adaptations made by a user running a manualtest scenario can lead to divergent routes.

In another example, a test scenario may be an automatic test scenariothat is interpreted and executed by a processor. The automatic testscenario may have a certain template (e.g., commands that execute asequence of test steps), and as such, executing the automatic testscenario should lead to a run of the test scenario that follows thetemplate route. However, there may be cases in which the automatic testscenario utilizes randomly generated or randomly selected values. Forexample, to automatically fill certain fields, the automatic testrandomly generates values, or selects certain random records from adatabase. Such an element of randomness which may introduce unexpectedvalues into a run of the automatic test scenario may lead to unexpectedbehavior of the system (e.g., generation of certain errors or executionof different transactions than were expected). Thus, a run of anautomatic test scenario may end up following a divergent route thatdiverges from its template route.

In yet another example, a semiautomatic test scenario, which is acombination of the aforementioned manual test scenario and automatictest scenario, is run by a user. The semiautomatic test scenario may endup not following its template route due various reasons, as describedabove. For example, actions of the user (e.g., values entered by theuser) may lead to unexpected behavior of the system. Similarly, valuesrandomly generated or automatically selected by the system may also leadto unexpected behavior while running the semiautomatic test scenario.Thus, a run of a semiautomatic may end up diverging from its intendedtemplate route.

A divergent route that diverges from a template route may do so invarious ways. A route analyzer may be configured to identify varioustypes of divergent routes, which diverge in different ways from thetemplate route.

In one embodiment, a route analyzer is configured to identify divergentroutes that diverge from the template route and later on converge backto the template route. FIG. 10A illustrates a template route 790 thatincludes a sequence of seven screens (SCR1 to SCR7) that are to beexecuted in that order. In one example, as illustrated in FIG. 10B, adivergent route 791 includes screens 1,2, and then two screens notincluded in the template route, screens 9 and 10, and then includesscreens 4,5,6, and 7 from the template route. Note that screen 3 fromthe template route is not part of the divergent route, but that need notbe the case in all examples. In other examples, a divergent route thatdiverges from the template route and later on converges back to thetemplate route may include all the elements of the template route and inaddition some elements that are not part of the template route.

In another embodiment, a route analyzer is configured to identifydivergent routes that diverge from the template route and do notconverge back to the template route. In one example, as illustrated inFIG. 10C, a divergent route 792 that diverges from the template route790 does not include screens 5,6, and 7 from the template route 790.Additionally, the divergent route 792 includes screens 9 and 11, whichare not part of the template route 790. Note that even though screens5,6, and 7 from the template route are not part of the divergent route,which need not be the case in all examples. In other examples, adivergent route that diverges from the template route and does notconverge back to the template route may include all the elements of thetemplate route and in addition some elements that are not part of thetemplate route that are to be executed after the elements of thetemplate route.

In yet another embodiment, a route analyzer is configured to identifydivergent routes that merge with the template route. A divergent route793 illustrated in FIG. 10D includes screens 8,9, and 12, which are notpart of the template route 790, and then four screens that are part ofthe template route 790 (screens 4,5,6, and 7). Note that screens 1,2,and 3 from the template route 790 are not part of the divergent route793, but that need not be the case in all examples. In other examples, adivergent route that merges with the template route may include all theelements of the template route and in addition some elements that arenot part of the template route that are to be executed before theelements of the template route and/or after it.

In one embodiment, a route from identified in a run of a test scenarios(i.e., the run follows the route), may be stored in a database.Optionally, the route may be a divergent route with respect to a certaintemplate route corresponding to a certain test scenario template.Optionally, storing the route involves recording the run and/orinformation extracted from the run in the database. Additionally oralternatively, storing the route may involve recording elements of theroute (e.g., test steps, transactions, and/or screens) in the database.Additionally or alternatively, storing the route may involve storing avalue computed from the run and/or the route, such as a hash valuecomputed from screen identification numbers and/or transactionidentification numbers. Optionally, storing a divergent route thatdiverges from a template route may involve storing information relatedto the template route and/or a test scenario template which defines thetemplate route. Optionally, storing a divergent route that diverges froma template route may involve storing information regarding one or moredifferences between the route and the template route.

In one embodiment, storing a route may involve taking note ofinformation pertaining to a certain user that ran a run which followsthe route, and/or a certain organization associated with the run. In oneexample, storing a route involves adding the certain user and/or thecertain organization to a list corresponding to the route. In anotherexample, storing a route involves incrementing a counter correspondingto a number of users and/or incrementing a counter that corresponds to anumber of organizations. Optionally, maintaining the aforementionedlists and/or counters may enable determination of popularity of certainroutes with users and/or organizations in general, and/or popularity ofcertain routes with specific users and/or specific organizations.

In one embodiment, a route stored in a database may be processed. Forexample, the route may be processed before being received by thedatabase, processed by the database (e.g., before storage and/or afterretrieval), and/or processed after being delivered from the database.

In one example, processing a route may involve selecting certain typesof elements to be included in the route. For example, processing a routemay involve retaining transactions included in the route, while othertypes of elements such as descriptions of screens are not stored.

In another example, processing a route may involve altering valuesincluded in elements belonging to the route. For example, data such asnumbers or addresses may be converted to a certain format. Additionallyor alternatively, values may undergo filtering, noise removal, and/orstemming of text. Altering values included in elements may assist instandardizing stored routes and make it easier to retrieve and/orcompare routes.

In yet another example, processing a route may involve filteringelements according to their popularity. For example, certain elementsthat are not utilized in a sufficient number of routes, by a sufficientnumber of users, and/or be associated with a sufficient number oforganizations may be excluded from a stored route. Low utilization mayindicate that the excluded elements are not likely to be useful forother organizations.

In still another example, processing a route may involve removingelements that are considered proprietary and/or removing values that maybe considered proprietary. For example, if screens included in a routeinclude proprietary field values, those values may be removed orreplaced with default values.

In some embodiments, similarity between two routes may be characterizedin different ways. For example, the two routes may be characterized asbeing “similar to each other” and/or “essentially the same”. Both termsindicate that the two routes have a degree of similarity between them(e.g., they have certain elements in common); however, as used herein,the term “essentially the same” typically indicates a higher level ofsimilarity between two routes compared to similarity indicated by theterm “similar to each other”. Usually, as used herein, any two routesthat are referred to as being essentially the same may also beconsidered similar to each other, but the converse is not necessarilytrue.

In one embodiment, two routes are essentially the same if they containthe same exact elements. For example, the two routes involve the samescreens (e.g., as determined by the fields in the screens and/or screenidentification codes). In another example, two routes that areessentially the same involve execution of the same transactions (e.g.,as determined from commands involved in the transactions and/ortransaction identifiers). Optionally, the two routes that areessentially the same may have different values associated with one ormore elements that are the same in both routes. For example, two routesmay be considered essentially the same if they contain the same screens(e.g., the screens may have the same identification numbers); however,different customizations done to elements of each route may causecertain field names and/or default values for fields to be slightlydifferent in the screens corresponding to the two routes.

In another embodiment, two routes may be considered similar to eachother and/or essentially the same if they involve similar elements. Forexample, the two routes involve the similar screens (e.g., as determinedby similar composition of fields in the two screens and/or similarvalues utilized by the two screens identification codes). In anotherexample, the two routes may involve execution of the similartransactions (e.g., as determined from similarity in commands involvedin the transactions and/or similar transaction identifiers).

Similarity between routes may depend on the number and/or proportion ofsame or similar elements shared by the routes. In one example, if tworoutes contain at least a first number of same elements, they may beconsidered similar to each other; and if the two routes contain at leasta second number of same elements, they may be considered essentially thesame. In this example, the second number may be higher than the firstnumber. For example, two routes may be considered similar if they bothinvolve executing 3 identical transactions, and they may be consideredessentially the same if they both involve executing 7 identicaltransactions. In another example, if two routes have at least a firstproportion of similar elements, they may be considered similar to eachother; and if the two routes contain at least a second proportion ofsimilar elements, they may be considered essentially the same. In thisexample, the second proportion may be higher than the first proportion.For example, two routes may be considered similar to each other if atleast 50% of the screens in a first rout of the two routes are similarto the screens in the second route of the two; furthermore, if at least80% of the screens in the first route are similar to the screens in thesecond route, the two routes may be considered essentially the same.

In yet another embodiment, similarity between two routes is determinedaccording to a distance metric that indicates that the distance(difference) between them. Optionally, if a distance between the tworoutes is below a first threshold, the two routes are considered similarto each other; and if the distance below them is below a secondthreshold, they are considered essentially the same. Optionally thesecond threshold is lower than the first. A distance metric may alsoexpress a level of similarity between routes. In such a case, ifsimilarity between two routes exceeds a first threshold, the two routesmay be considered similar to each other, and if the similarity exceeds asecond threshold, possibly higher than the first, the two routes may beconsidered essentially the same.

In one example, routes are represented as a sequence of transactionidentifiers. A distance metric between two routes may be a standardsequence similarity metric, such as edit distance or Manhattan distance.In this example, two routes may be considered essentially the same ifthe edit distance and/or the Manhattan distance is below a predeterminedthreshold; for instance, the predetermined threshold may be 20% of thelength of the longest sequence corresponding to one of the routes.

In another example, routes may be converted to a vector representation.For example, each element may correspond to one or more dimensions in avector space. Optionally, certain vector dimensions may correspond tothe presence of certain elements (e.g., a value of 1 is given if theelement is present in a route and 0 otherwise). Optionally, certainvector dimensions may correspond to values associated with elements inthe routes. Those skilled in the art may recognize that there arevarious ways in which a route may be represented as a vector. The choiceof representation may depend on various factors, such as the length ofthe desired vectors and/or the type of data included in routes. In oneexample, two routes may be considered essentially the same if a distancebetween vectors representing the two routes is below a predeterminedthreshold. The cosine of the angle between two vectors may be used tomeasure the distance (or similarity) between the vectors. For example,if the angle between the two vectors, as determined from the cosine, issmaller than a predetermined threshold, the two routes corresponding tothe vectors may be considered essentially the same. Optionally, if theangle is below a second, slightly higher predetermined threshold, thetwo routes may be considered similar to each other.

Configuration Elements and Changes

In one embodiment, the software systems may include many modules thatcan be configured to suit an organization's needs. Configuring a systemmay involve various aspects of a software system, modules belonging tothe software system, and/or business processes run on the softwaresystem. Optionally, in some embodiments, configuring a system is done,at least in part, using configuration files, setup file, and/orcustomization code that are read by the system and may be altered by anorganization to suit the organization's needs. In one example,configuring a SAP ERP system involves entering, editing, and/or removingvalues from configuration tables. Optionally, the tables may definevarious aspects of fields (e.g., name and type) for certain data tablesusers may access via the ERP system. In another example, an Oracle™database system is configured using Oracle form personalization, whichis a standard feature provided by Oracle™ and supported by Oracle™ tocustomize the forms. Forms personalization enables changing propertiesof fields, such as hiding fields, making fields mandatory, creating zoomfunctionality, and/or dynamic changes of lists of values. In oneembodiment, customizations refer to user-generated code, such as “userexit” in SAP, and customization code in Oracle.

In some embodiments, a configuration element is one or more details(e.g., a line in a configuration file, a field name, a function name),which can be used to configure behavior of a software system. In oneexample, a configuration element may define certain aspects of datastructures used by the system (e.g., column names in a table) and/ortype of values in a column belonging to a table. In another example, aconfiguration element may determine behavior of a system, such asdefining what functions should be called under certain conditions (e.g.,what to do with a printing job).

In one embodiment, a change to a configuration element may be referredto as a “configuration change”. Optionally, a configuration change mayinclude one or more actions that involve adding a configuration element,deleting a configuration element, and/or editing a configurationelement. Optionally, a configuration change may require running one ormore test scenarios in order to verify that the configuration changecaused a desired effect and/or in order to verify that the configurationchange did not cause an undesired effect.

In one embodiment, a configuration element may have an associated value.For example, a configuration element called “default field width” mayhave an associated value 20. Optionally, a change to an associated valueof a configuration element may be referred to as a “configurationchange”. Optionally, a configuration change may involve adding anassociated value to a configuration element, deleting an associatedvalue from a configuration element, and/or editing an associated valueof a configuration element.

In some embodiments, there may be various ways in which configurationelements and/or configuration changes may be identified. Identifying aconfiguration element enables a software system and/or a user todetermine what configuration element is being referred to. Similarly,identifying a configuration change enables the software system and/or auser to determine what configuration element being changed and/or whatchange is done to an associated value of the configuration element.

In one example, a configuration change is identified by a code, such asa hash code or an identification number. Optionally, the code alsoidentifies an associated value of the configuration element and/or achange to an associated value of the configuration element. In anotherexample, a configuration change and/or a configuration element may becharacterized by a new configuration file (e.g., a new setup file forthe system). Optionally, a comparison between a new configuration fileand a previous configuration file may characterize configurationchanges, that involve changes to configuration elements and/orassociated values of configuration elements.

In one embodiment, configuration elements and/or configuration changesmay be clustered into clusters of similar configuration elements and/orconfiguration changes. Optionally, clusters of similar configurationelements include configuration elements that deal with the same, orsimilar, system modules, business processes and/or database tables.Optionally, clusters of similar configuration changes includeconfiguration changes that deal with the same, or similar, configurationelements. Optionally, clusters of similar configuration changes includeconfiguration changes that are associated with similar values.

In one embodiment, configuration elements and/or configuration changesare clustered utilizing text clustering approaches. For example,clustering is performed on configuration files that include theconfiguration elements and/or configuration changes. Those skilled inthe art may identify various text clustering algorithms that may beutilized to cluster configuration elements into clusters of similarconfiguration elements. Optionally, configuration elements and/orconfiguration changes may be considered similar if configuration filesthat include the configuration elements and/or the configuration changesare considered similar (e.g., they belong to a same cluster).

In another embodiment, clustering configuration elements and/orconfiguration changes may be done based on values associated with theconfiguration elements. Optionally, clusters of similar configurationelements and/or configuration changes may include same or similarconfiguration elements and/or configuration changes, which have similarassociated values in several ways. In one example, portions ofconfiguration files that include multiple configuration elements and/orconfiguration changes are converted to vector representation; eachvector dimension corresponds to a certain configuration element and thevalue entered in the vector dimension corresponds to an associated valueof the certain configuration element. Those skilled in the art mayrecognize various algorithmic approaches that may be utilized to clustervectors representing the portions of the configuration files (e.g.,k-means or hierarchical clustering).

Clustering Runs of Test Scenarios

In some embodiments, runs of test scenarios may be clustered. Clusteringthe runs may involve assigning a run of a test scenario to at most onecluster (e.g., “hard clustering” or partitioning). Alternatively, theclustering may involve assigning a run of a test scenario to one or moreclusters. For example, the clustering may be “soft clustering” in whicha run of a test scenario may belong to various clusters, possibly withdifferent probabilities or levels of association to each cluster.Optionally, clusters of runs contain runs that are similar to eachother.

In one embodiment, runs of test scenarios may be partitioned intoclusters based on one or more values from the runs of test scenarios.For example, runs that involve a same start and/or end test step may beplaced in the same cluster (e.g., runs that start from the same screenID and end with an error are placed in the same cluster). In anotherexample, runs that have a certain field (e.g., customer bank accountnumber) are placed in the same cluster.

Clusters of runs of test scenarios may have different characteristics indifferent embodiments. In one embodiment, a cluster of runs of testscenarios should be of a size that reaches a predetermined threshold.Optionally, the predetermined threshold is greater than one. Forexample, each cluster should contain at least 3 runs. Optionally, thepredetermined threshold is proportional to the number of runs beingclustered. For example, each cluster may be required to contain at least0.1% of the runs of test scenarios being clustered.

Clusters of runs may also be constrained according to the source of theruns belonging to the clusters. In one embodiment, a cluster of runs oftest scenarios must include runs of at least a first predeterminednumber of different users and/or of users belonging to at least a secondpredetermined number of organizations. Optionally, the firstpredetermined number is greater than one and/or the second predeterminednumber is greater than one. In one example, the first predeterminednumber is 10, and the second predetermined number is 2; thus, eachcluster includes at least 10 runs, and not all those runs are associatedwith the same organization. In another example, the first predeterminednumber is 5, and the second predetermined number is 5; thus, eachcluster needs to contain runs associated with at least 5 organizations.

In one embodiment, ensuring that clusters of runs have certaincharacteristics, such as a certain size and/or include runs of a certainsource, is done by a clustering algorithm that generates the clusters.For example, the clustering algorithm may ensure that each clusterincludes runs of test scenarios of at least a predetermined number ofusers. Alternatively or additionally, ensuring that clusters of runshave certain characteristics may be done after clustering. For example,after clusters are generated, they may be filtered to remove clustersthat have a size that is smaller than a predetermined threshold.

In one embodiment, clustering of runs of test scenarios may involveprocedures that rely on some runs being similar based on a criterion ofsimilarity. Optionally, a cluster of runs may include similar runs.Optionally, by similar runs it is meant that two runs are similar toeach other according to the criterion. Optionally, by similar runs it ismeant that at least a certain percentage of the runs belonging to acluster are similar to each other according to the criterion. Forexample, a cluster may be considered to include similar runs if 90% ofthe pairs of runs in the cluster are similar according to the criterion.Optionally, by similar runs it is meant that runs belonging to thecluster are all similar to a representative of the cluster, such as oneof the runs belonging to the cluster or an average run of the cluster(e.g., a centroid of the cluster).

The criterion according to which similarity between runs may beestablished, may have several forms. For example, the criterion forsimilarity between runs may be that similar runs include at least oneof: essentially the same fields, similar combinations of fields, similarexecution of transactions, similar user interactions, similar requests,similar test steps, and/or similar calls to procedures. Optionally,similarity between runs of test scenarios may be determined based onproperties of their respective test scenarios of which the runs areinstantiations; for example, by comparing the test steps used in eachtest scenario.

In one example, various runs of essentially the same test scenario(e.g., essentially the same testing script) are considered similar. Inanother example, runs of different test scenarios (e.g., using slightlydifferent testing scripts), may be considered similar if certainsimilarity criteria are met (e.g., similar fields types or names,similar field values, similar screen content and/or layout, and/orsimilar return values in the runs). In yet another example, runs of testscenarios are considered similar, if the test scenarios from which theywere instantiated are similar (e.g., involve similar screens, similarfields, and/or similar field values).

In one example, at least part of the runs of test scenarios involvesoftware that runs on remote servers such as cloud-based servers.Monitoring a user running a test scenario may involve monitoring thecontent of the network traffic, such as information exchanged between aninput and/or output device of the user and a cloud-based server. In thisexample, runs of test scenarios may be considered similar if the networktraffic associated with them is similar (e.g., it follows a similarexchange sequence, and/or content of the network traffic is similar).

In one embodiment, clustering runs of test scenarios to clusters thatinclude similar runs may be based on counting the number of similarfields used in corresponding screens that are included in the testscenarios; the larger the number of similar fields in the testscenarios, the more similar the runs of the test scenarios areconsidered to be. Optionally, fields may be considered similar if theyinclude the same type of data. Additionally or alternatively, fieldsthat have the same values are considered similar to each other. In oneexample, fields that include both the same type of data and the samevalues are considered more similar to each other than fields that havethe same type of data (but different values).

In one embodiment, a type of data of a field is determined according tothe context of the field in the screen. For example, if a field ispreceded on a screen by the words “amount” or “sum due”, the field isconsidered to be numerical; while if the field is preceded on a screenby the words “address” or “ship to”, it is considered to be a string.Additionally or alternatively, the type of data of a field may bedetermined according to the features of the data structure representingthe data. For example, if the field stores data in a variable that holdsintegers or floating-point values, the type of data is considered to benumerical. However, if the field stores data in an array, or vector ofcharacters, the data type is considered to be a string. Additionally oralternatively, the type of data of a field may be determined accordingto meta data associated with a screen or database associated with thefield. For example, a description of a column, in a database table whichis to receive data from a field may be indicative of the type of data.In another example, meta data tags (e.g., XML tags) associated with ascreen may indicate the type of data.

In another embodiment, clustering of runs of test scenarios to clustersthat include similar runs of test scenarios may be based on similaritybetween orders of displaying similar fields in corresponding screens.The closer the order of presentation of similar fields in testscenarios, the more similar runs of the test scenarios are considered tobe. In one example, test scenarios are represented, at least in part, asa sequence of field types, and/or fields IDs. The similarity between tworuns of test scenarios may be determined to be inversely proportional tothe number of editing steps that need to be taken to transform onesequence of identifiers to another (“edit distance”); the lower the editdistance between representations of two runs, the more similar they areconsidered to be (and so are their corresponding runs).

In yet another embodiment, clustering of runs of test scenarios toclusters that include similar runs may be based, at least is part, onsimilarity between executed procedures that are described in the runs.For example, the larger the overlap in the corresponding sets ofprocedures performed by each test scenario, the more similar runs of thetest scenarios are considered to be. Optionally, the clustering may befurther based on the order of the execution of the procedures; thecloser the order of execution of procedures in different test scenarios,the more similar runs of the test scenarios are considered to be. Incases in which test scenarios involve execution of essentially the sameprocedures in essentially the same order, the similarity between runs ofthe test scenarios may be considered to be high.

In still another embodiment, the clustering of runs of test scenarios toclusters that include similar runs of test scenarios may be based ontransactions described as executed in the runs of the test scenarios.For example, test scenarios may be characterized by the individualcommands run by the system as part of the test scenario. Runs of testscenarios in which similar commands are executed may be consideredsimilar for the purpose of clustering. Optionally, a cluster of runsthat contains runs that have a certain proportion of common transactionsexecuted in all runs in the cluster is considered a cluster of similarruns. For example, if at least 50% of the transactions involved in eachrun in a cluster appear in all other runs in the cluster, the cluster isconsidered to be a cluster of similar runs.

Logged activities related to running test scenarios may also be utilizedfor the purpose of clustering and/or determining similarity between runsof test scenarios. For example, clustering of runs of test scenarios toclusters that include similar runs may be based on one or more of thefollowing logged activities: a list of users who ran the test scenarios,an analysis of access to a database, messages returned from the executedtransactions (e.g., valid, warning, or error messages), fields whichreturned values in the transactions, and/or procedures utilized by thetest scenario (e.g., as identified by logs of run time analysis). Theclustering may be done according to there being similarity, involvingone or more of the aforementioned logged activities, between testscenarios whose runs are assigned to the same cluster. Optionally,logged activities may be represented as feature values that may be putin a vector corresponding to a run. For example, if a certain activityis performed during a run, a vector corresponding to the run has 1 in acertain position, and otherwise there is a 0 in the certain position.

Similarity of runs of test scenarios may be determined, in someembodiments, according to the test scenarios and/or templates from whichthe runs were instantiated. Optionally, similarity of the test scenariosand/or templates may define similarity of the runs that wereinstantiated from the test scenarios and/or templates; thus, runs thatwere instantiated from similar test scenarios and/or templates areplaced in the same clusters (e.g., by the clustering module performingthe clustering of the runs). Alternatively, similarity of the testscenarios and/or templates may be used as features that assist indetermining similarity of runs.

In one embodiment, runs that were instantiated from the same testscenarios and/or the same templates may be considered similar.Optionally, two test scenarios and/or two templates are considered thesame if they involve execution of the same test steps, screens, and/ortransactions. In one embodiment, two test steps are considered the sameif they perform the same exact task and include the same exactassociated data, while in another embodiment the two test steps areconsidered the same if they perform the same task but possibly involvingdifferent associated data. For example, a first test step that involvesentering a new product that is a screwdriver (with details relevant to ascrewdriver) may be considered in the latter embodiment as being thesame as a second test step that involves entering a new product that isa hammer (with details relevant to a hammer); however, according to theformer embodiment, the first and second test steps may not be the same.Similarly, screens that include field names and field values may beconsidered the same in one embodiment if the field names and the fieldvalues are the same; in another embodiment, the screens may beconsidered the same if the field names are the same. The same logic mayalso be applied to transactions; in some examples transactions may beconsidered the same if they are completely identical, while othertransactions may be considered similar if they include some of the sameand/or similar elements (e.g., the transactions involve similarscreens).

In another embodiment, similarity of test scenarios and/or templates isdetermined by comparing and/or counting similar elements in the testscenarios and/or templates. If the number and/or proportion of thesimilar elements reaches a predetermined threshold, then runsinstantiated from the test scenarios and/or templates may be consideredsimilar and placed by clustering in the same cluster of runs. Forexample, if more than 50% of the screens included in two templates aresimilar, then runs instantiated from the two templates may be placed byclustering in the same cluster.

In yet another embodiment, test scenarios and/or templates from whichruns were instantiated may be utilized to generate feature values, whichare used to determine similarity of the runs to each other. For example,a vector of features representing a run may include values extractedfrom a template and/or test scenario from which the run wasinstantiated.

Clustering of runs of test scenarios to clusters of similar runs may bebased on data associated with the runs. Such data may include datadescribing conditions under which a run was executed. For example, thedata may describe aspects of the system (e.g., data involving modules,hardware, and/or software versions). In another example, such data maypertain to a test runner, such as the role (or assumed role) of thetester in an organization, level of skill of the tester, and/orpermissions granted to the tester.

In some embodiments, clustering of runs of test scenarios to clusters ofsimilar runs is done, at least in part, according to descriptionsrelated to the systems on which the test scenarios were run. Forexample, such descriptions may include configuration elements (e.g.,configuration files, customization code and/or setup files).Additionally or alternatively, the descriptions may includeconfiguration changes (e.g., addition, deletion, and/or modifications)to the configuration elements. Thus, for example, runs of test scenariosconcerning modules with similar configuration files (e.g., thecustomization files indicate similar default procedures and/or databaseaccesses) may be placed in the same cluster. In another example, runs oftest scenarios executed as a response to similar configuration changes(e.g., as determined by the original and/or changed values involved inthe configuration changes), may be placed in the same cluster based ontheir similar configuration changes.

In some embodiments, runs of test scenarios may be represented asvectors of features that may be converted to numerical values. Forexample, certain dimensions in the feature vectors may correspond to thepresence or absence of certain fields, procedures, test steps, and/ortransactions in a test scenario (e.g., a value of ‘1’ is given to afeature if a field has a certain value in the test scenario, and ‘0’otherwise. Alternatively, a value of ‘1’ is given to a certain featureif a certain procedure is called in the test scenario, and ‘0’otherwise). In another example, certain dimension in the feature vectorscontain values of a certain field from a run (e.g., time, data, orprice), or are derived from processing one or more field values (e.g.,averaging the delivery time from multiple entries of individual deliverytimes entered in a screen). In yet another example, certain values in afeature vector are assigned numerical values according to categories towhich values from the runs belong. For example, a sale may becategorized as “domestic” or “foreign”, and accordingly be given a valueof “1” or “2” in the feature vector.

Feature vector representations may be utilized in order to compute adegree of similarity between feature vectors of runs of test scenarios.For example, in cases where the feature vectors contain numerical values(or can be converted to numerical values), the distance similaritybetween vectors representing test scenarios may be computed using one ormore of the following established distance metrics: Euclidean distanceof various norms, vector dot product, cosine of angle between vectors,Manhattan distance, Mahalanobis distance, Pearson correlation, andKullback-Leibler divergence.

In one embodiment, a cluster of similar runs includes runs that arerepresented by similar vectors. Optionally, similar vectors may becharacterized in various ways. In one example, similar vectors arevectors whose average pairwise similarity is above a predeterminedthreshold (e.g., the threshold may be 0.5). Optionally, the averagepairwise similarity is determined by computing the average of the dotproduct of each pair of vectors. In another example, similar vectors arevectors that are all similar to a certain representative vector; e.g.,the vectors all within a sphere of a certain Euclidean distance from therepresentative.

Those skilled in the art may recognize that various clusteringalgorithms and/or approaches may be used to cluster runs of testscenarios into clusters that include similar runs of test scenarios. Forexample, the clustering may be done using hierarchical clusteringapproaches (e.g., bottom-up or top-down approaches) or usingpartition-based approached (e.g., k-mean algorithms). In addition, someof the test scenarios may have assigned clusters while others may not.In such a case, a semi-supervised clustering approach may be used suchas an Expectation-Maximization (EM) algorithm.

In one embodiment, the clustering of the runs of test scenarios toclusters that include similar runs may be done utilizing a classifierthat is trained to assign test scenarios to predetermined classes.Optionally, the classifier is trained on labeled training data thatincludes training data that includes representations of runs of testscenarios (e.g., feature vectors) and labels corresponding to clustersto which the runs are assigned. If the labels in the training data areassigned according to some (possibly arbitrary) notion of similaritybetween test scenarios, clusters of test scenarios that have the samelabel assigned by the classifier are likely to contain runs that aresimilar according to the notion of similarity.

Optionally, runs of test scenarios are labeled according to the modulethey involve (e.g., “sales”, “human resources”, “manufacturing”), thetype of activities involved in the scenario (e.g., “accessing database”,“data entering”, “report generating”), fields in the test scenario(e.g., “customer ID”, “part number”), properties of the test scenario(e.g., “fast”, “many steps”, “expensive”, “includes private data”),and/or results of a run of the test scenario (e.g., “ok”, “error”,“access denied—permissions”). Optionally, labels may be constructed frommultiple attributes. For example, a run of a test scenario may belabeled as “involve sales, no private data, access corporate database”.

Optionally, labels assigned to runs of test scenarios may be generatedand/or assigned manually (e.g., by a tester running a test), and/orautomatically, e.g., by a procedure that analyzes a test scenario todetect attributes describing it (e.g., what modules and/or procedures itinvolves).

Those skilled in the art may recognize that there are many algorithms,and/or machine learning-based approaches, that may be used to train aclassifier of runs of test scenarios using labeled training data. Forexample, some examples of the algorithms that may be used includelogistic regression, decision trees, support vector machines, and neuralnetwork classifiers.

In some embodiments, clusters of runs of test scenarios may be assigneda cohesion rank that describes how close to each other are runsbelonging to the cluster. A cohesion rank of a cluster may describevarious aspects related to the closeness of runs of test scenariosbelonging to a cluster. In one example, similarity of runs of testscenarios is related to the fields included in the screens of the testscenarios; some of the fields may be common to the screens involved inruns belonging to the cluster, and some may not. The cohesion rank maybe proportional to the number of fields that are common in the runsbelonging to the cluster (e.g., they belong to at least 50% of the runsin the cluster). The cohesion rank can be between two runs of testscenarios or between groups of runs of test scenarios. Optionally, thecohesion rank may be expressed via cluster metrics such as averagedistance from the cluster centroid or the ratio between the averageradius of a cluster (intra-cluster distance) and the average distancebetween clusters (inter-cluster distance).

Clusters generated in the embodiments (e.g., clusters of runs of similartest scenarios) may be filtered in various ways. Optionally, thefiltering may be done in order to reduce the number of clusters thatneed to be considered and/or retain clusters with a certain desirableproperty. Optionally, a test scenario template generated from a filteredcluster and/or representing a filtered cluster is not suggested to auser. Alternatively or additionally, a filtered cluster is not utilizedfor generating a test scenario template and/or does not have a testscenario template to represent it. Optionally, clusters are filteredaccording to their size. For example, clusters of runs of test scenariosthat contain less than a predetermined number of runs are filtered.Optionally, clusters are filtered according to the number of theirassociations. For example, runs of test scenarios in a cluster may beassociated with certain organizations (e.g., each run may be associatedwith an organization). Clusters containing runs that are associated withtoo few different organizations may be considered too homogenous andfiltered.

Test Scenario Templates

A test scenario template may include various test steps that need to beperformed by a user in order to test certain aspects of a system beingtested. Optionally, in some embodiments, a test scenario template may belacking one or more values that need to be provided in order to run atest scenario based on the test scenario template. In such a case, auser running a test scenario based on the template may be required toenter the one or more values that need to be provided, in order tocomplete the template of the test scenario; alternatively, the one ormore values and be calculated and/or guessed. In one example, a testscenario template may describe a test scenario in which an order processis completed; the template may be lacking a customer name and productID, and in order to run an instantiation of the template, a user may berequired to enter the lacking details. In another example, a templatemay be of a test scenario in which rush shipping is performed; a screenpresented to the user may already have the rush shipping box checked,but the user may still need to add details such the client accountnumber and shipping address.

In one embodiment, a template may include certain values from which auser may be required to choose in order to run a test scenarioinstantiated from the template. For example, a template may have variousvalues for a discount (e.g., 10%, 20%, or 50%); a user needs to choosefrom in order to complete a transaction that is part of the template.Optionally, the values the user may need to choose from are derived fromone or more runs of test scenarios that were used to generate thetemplate. For example, the aforementioned values of the discount are themost popular values found to be used in runs of test scenarios fromwhich the template was generated.

In some embodiments, one or more runs of test scenarios may be used inorder to generate a test scenario template to represent them.Optionally, the test scenario template is generated by a templategenerator module. In one example, the runs of test scenarios may belongto a cluster. Optionally, the test scenario template may identify, forits corresponding cluster, one or more transactions used by testscenarios belonging to the cluster. Additionally or alternatively, thetest scenario template may identify a way of using transactions that arepart of the test scenarios whose runs belong to the cluster. Forexample, a template may define the order of programs that are to becalled, the type of actions that are to be taken by a user (e.g., whatfields to fill, what buttons to push), and/or provide default values toat least some of the fields (e.g., enter default values to some fieldsthat appear on screens that are part of test scenarios generated fromthe template). Additionally or alternatively, a test scenario templatemay identify one or more possible values that may be used to run atransaction identified by the template.

It is to be noted that phrases such as “identifies a transaction”involves any type of transaction identification, such as a transactionidentification number, a name of a transaction, a description of atransaction, a screenshot of a transaction, computer code of atransaction, and/or any other method that enables a human and/or acomputer to link between a possible value and a transaction.Additionally, identifying a transaction may involve noting a transactionidentification. In one example, noting a transaction identificationinvolves storing the transaction identification; for example, a templatemay store transaction identification numbers that are to be executedwhen running an instantiation of the template. In another example,noting a transaction identification involves providing the transactionidentification. For example, upon a query, scanning a template maygenerate and/or transmit names of transactions that are stored in it.

Generating the template may involve utilizing information from aplurality of runs of test scenarios. For example, one or more teststeps, commands, and/or values may be copied from at least one of theplurality of runs of the test scenarios and placed in the template.Additionally or alternatively, information contained in at least some ofthe plurality of runs may be analyzed in order to determine certain teststeps, commands, and/or values are to be used in the template. Note thatnot all information in the plurality of run of the test scenario needsto be utilized to generate the test scenario template. For example, atest scenario template may include a proper subset of test stepsincluded in certain runs. In another example, certain runs, such ascertain runs in a cluster, may be disregarded when generating a templatebased on other runs in the same cluster.

In some embodiments, template generation may be based on informationobtained from one or more test scenarios. Optionally, instantiations ofthe one or more test scenarios are runs of test scenarios obtainedand/or identified from activity data of users. In one example, a testscenario may include a script comprising test steps, and one or moretest steps from the script are included in the template. Note that thescript may be one various types of media; for example, a hard copydocument (e.g., paper), an electronic document (e.g., a Microsoft Word™or a PDF document), and/or a list of computer commands (e.g., a scriptfor an automatically run test scenario). In another example, a testscenario may include certain values that may be altered by a userrunning instantiations of the test scenario (e.g., the certain valuesmay be default values). In this example, to generate the template it maybe desirable to utilize a certain value from a test scenario, ratherthan a user-provided value from a run that is an instantiation of thetest scenario.

A test scenario template may be generated based on data coming fromseveral sources. In one embodiment, a template is based on automatictest scenarios (e.g., scripts run automatically by a program without anyessential human intervention). Additionally, the template may also bebased on the runs of the same test scenario. Thus, certain values in thetemplate may come from the test scenarios, while other values may comefrom the runs. In another example, a template may be based on runscoming from different organizations. The template may contain certainscreens coming from a first organization (e.g., as they appeared in runsassociated with the first organization), while other screens in thetemplate may come from runs associated with a second organization.

In one embodiment, a test scenario template is generated from a firstrun of a test scenario run by a first user and a second run of a testscenario run by a second user belonging to a second organization.Optionally, the first and second runs were obtained from monitoring ofthe first and second users, respectively. Additionally, the firstorganization may be different from the second organization and the firstuser is not the second user. The test scenario template generated fromthe first and second run identifies a transaction used in the first andsecond runs and one or more possible values for running the transaction.Optionally, additional runs of test scenarios, besides the first andsecond runs, are used to generate the test scenario template.

In one embodiment, at least one of the possible values for running atransaction in a template generated from first and second runs may bederived from values obtained from the first and/or second runs.Optionally, the at least one of the possible values for running thetransaction does not appear in each of the first and second runs. Forexample, the at least one of the possible values is an average of afirst value from the first run, and a second value form the second run,and the first value does not equal the second value. Optionally, thetest scenario template includes a combination of the possible valuesthat may be used to run the test scenario template, and the combinationdoes not appear in any of the first and second runs. For example, acombination of possible values involves fields f₁ and f₂; the first runhas values v₁ and u₁ for the fields f₁ and f₂, respectively, and thesecond run has values v₂ and u₂ for the fields f₁ and f₂, respectively.In addition, v₁ does not equal v₂ and u₁ does not equal u₂. In thisexample, if the test scenario template has a value v₁ for f₁ and u₂ forf₂, then it contains a combination of possible values that does notappear in any of the first and second runs.

In one embodiment, the first and second runs upon which a template isbased, may each be manual, semi-automatic, or automatic runs of testscenarios. In one example, a test scenario template is generated from afirst run of a first test scenario which is a manual test scenario and asecond run of a second test scenario which is an automatic testscenario. In another example, a test scenario template is generated froma first run of a first test scenario and a second run of a second testscenario, and both the first and second runs are automatic.

In another embodiment, the first test scenario and second test scenarioare the same test scenario. Thus, the first run and the second run areinstantiations of the same test scenario. In this case, despite beingruns of the same test scenario, the first and second runs may bedifferent (e.g., due to different inputs provided by a user during theirrunning). Alternatively, the first test scenario and second testscenario may be different test scenarios. Thus, the first run and thesecond run are runs of different test scenarios; however, the first andsecond runs may be similar due to similarities (despite being different)between the first and second test scenarios and/or similarities ininputs provided by the user while running the first and second runs).

In one embodiment, generating a test scenario template involvesreceiving multiple runs (e.g., runs belonging to a cluster of similarruns), and selecting at least a first run and a second run, from amongthe multiple runs, upon which the template is to be based. Optionally,the first and second runs are selected such that they belong are runs ofdifferent users and/or runs of users belonging to differentorganizations. Optionally, the first and second runs are runs thatexhibit, on average a high similarity to the multiple runs (e.g., theyare similar to a cluster centroid). Optionally, the first and secondruns are selected such that other of multiple runs have a similarity toeither the first run or the second run that reaches a predeterminedthreshold. Optionally, the first and second runs are selected accordingto a profile, such as a profile of a certain user. Optionally, theprofile indicates transactions and/or values typically utilized by thecertain user, and the first and second runs that are selected involvetransactions and/or values that appear in the profile.

In one embodiment, generating a test scenario template from a cluster ofruns of test scenarios involves identifying a representative run of atest scenario for the cluster, and using the representative run of atest scenario as basis for the at least one template.

In one embodiment, generating a test scenario template from a run of atest scenario may involve utilizing information from the run of the testscenario. For example, one or more test steps, commands, and/or valuesmay be copied from the run and placed in the template. Additionally oralternatively, information contained in the run may be analyzed in orderto determine certain test steps, commands, and/or values are to be usedin the template. Note that not all information in the run of the testscenario needs to be utilized to generate the test scenario template.For example, the template may include a proper subset of test stepsincluded in the run of the test scenario. Optionally, certaininformation in the run which may be deemed proprietary is not utilizedfor the template. Additionally or alternatively, certain information inthe run which may be deemed proprietary is removed from the template.

In another embodiment, generating a test scenario template from runs oftest scenarios (e.g., the runs belong to a cluster of similar runs oftest scenarios), may involve utilizing values of a field, as they appearin runs, in order to set the value of the field in the test scenariotemplate.

In one embodiment, the value for the field in the test scenario templateis selected from the values of the field in the runs of test scenariosin the cluster. For example, the value of the field in the template isset according to the value of the field in a randomly selected run of atest scenario from the cluster.

In one embodiment, the value for the field in the test scenario templateis generated by a function with one or more parameters that are setaccording to the values of the field in the runs of test scenarios inthe cluster. Optionally, the value generated by the function is not avalue found in any of the runs belonging to the cluster. For example, inorder to fill the field “age” in the template, a distribution of thevalues of “age” in runs in the cluster may be learned, and then thevalue in the template may be set according to the expectation of thelearned distribution or randomly drawn according to the distribution. Inthis example, the resulting value of “age” may not be an actual valuethat appears in any of the runs belonging to the cluster.

Test scenario templates may be suggested to a user so that the user mayrun an instantiation of the templates on a system to which the userbelongs and/or is associated with. Optionally, a template may begenerated from one or more runs of test scenarios associated withvarious organizations to which the user does not belong.

In some embodiments, a user is suggested to run a test scenario templateby presenting the user with a suggestion on a user interface. Forexample, the user interface may be a screen (e.g., monitor, imageprojected on a screen, and/or augmented/virtual reality display). Inanother example, the user interface may involve a speaker. Optionally,the user may interact with a system via the user interface in variousways; for example, touch (e.g., via a touch screen), typing (e.g., on aphysical and/or virtual keyboard), a hand-operated control device (e.g.,a mouse), and/or voice commands. Optionally, the user is given anopportunity to modify via the user interface, data related to theinstantiation and presented on the user interface; for example, the usermay modify default values, set according to the template, and presentedon a screen that is presenting while running an instantiation of thetemplate. Optionally, the user interface may initiate an instantiationof a test scenario template; for example, the user interface may presenta first screen of the test scenario template and prompt the user to takea certain action to advance execution of the template.

In one embodiment, a user is recorded while running a certain testscenario. If it is identified that the certain test scenario is similarto a test scenario template (e.g., the certain test scenario involvessimilar test steps, transactions, and/or values as those involved in thetemplate), it may be suggested to the user to run an instantiation ofthe test scenario template. Optionally, a suggestion to the user to runan instantiation of the template is made via a user interface.

In another embodiment, a profile of a user is utilized to suggest to theuser to run a test scenario template appropriate to the user based oninformation included in the profile. In one example, the profile mayinclude various information regarding the user and/or an organization towhich the user belongs, such as information regarding modules, businessprocesses, and/or transaction utilized by the user and/or organization.In another example, the profile may include runs of the user and/orusers belonging to a certain organization, and/or data derived from theruns. In addition to the profile, and/or instead of it, the suggestionof a template to the user may be done according to other factors such ascoverage of templates, importance of templates, ranking of templates,and/or connectivity factors of templates, as described in more detailbelow.

Coverage

In order to validate that certain system elements operate correctlyafter installation, customization, a change, and/or an upgrade is doneto the system—it is often the case that many test scenarios need to berun. In order to validate the system's performance, it is desirable thatthe test scenarios that are run should cover many elements that may berelated to, and/or affected by, the installation, customization, change,and/or upgrade. For example, the elements may include varioustransactions that may exhibit undesired behavior due to the certainconfiguration change and/or upgrade; each executed test scenario may beable to be used to test a subset of the transactions that are likely tobe affected by the change, and/or update, in order to determine ifunwanted system behavior occurs that involves one or more transactions.Therefore, the coverage offered by a set of test scenarios (or testscenario templates), such as which modules, business processes, and/ortransactions are evaluated by running a set of test scenarios, becomesan important consideration when selecting which test scenarios to run.Ideally, it would be desirable to obtain as large coverage as possiblewith test scenarios, however, often restrictions stemming from limitedresources for running test scenarios have also to be taken into account.

In one embodiment, coverage refers to a percent of business processes,used by an organization, that are sufficiently tested by test scenariosin relation to the total number of business processes used by anorganization or a certain user. For example, if 40% of the businessprocesses run by users of an organization are sufficiently tested bycertain test scenarios, then the coverage of the certain test scenariosis 40%. In some cases in order for a module, business process, and/ortransaction to be sufficiently tested, more than one aspect of themodule, business process, and/or organization needs to be tested. Thismay involve running multiple tests in order to cover the more than oneaspect. For example, different combinations of input data for the samescreen need to be provided in order to test various aspects of a certainbusiness process.

In one embodiment, a required coverage for a certain user that runs testscenarios, is received and utilized for suggesting test scenariotemplates for the user. For example, the required coverage may listcertain transactions that need to be evaluated with test scenarios runby the user, and optionally the number of test scenarios that need to berun to evaluate at least some of the transactions. Alternatively oradditionally, the required coverage may indicate what percentage of asystem's transactions need to be evaluated by the test scenarios. Inanother example, the required coverage may indicate which businessprocesses, screens, and/or specific fields need to be evaluated byrunning test scenario.

In one embodiment, achieving a required coverage is done by evaluating alarge number of test scenario templates and estimating coverage achievedby individual templates and/or subsets of templates. Following theevaluation, a subset of templates that includes at least one template issuggested to the user in order to achieve the required coverage.Optionally, the suggested subset that includes at least one template isa subset with an essentially minimal number of templates and/or involvesperforming an essentially minimal number of test steps. It is to benoted that by an “essentially minimal” number it is meant a number closeto the absolute minimal possible, for example up to 10% more than theabsolute minimum. In some cases, the absolute minimal number oftemplates may be determined by an exhaustive evaluation of all possiblesubsets of templates; however, this may prove intractable if a largenumber of templates are involved.

In one embodiment, calculating the required coverage for a certain useris based on a usage information of a certain user. For example, byanalyzing the usage information a list may be prepared of certainmodules, business processes, and/or transactions the user frequentlyutilizes. This list can then be used to guide a search for a certainsubset of test scenario templates that tests aspects of items on thelist which need to be tested.

Finding a minimal sized subset of template obtaining a desired coverageneed not require exhaustive search in every case. Those skilled in theart may recognize that there are structured methods for evaluating asearch space of subsets of templates such as branch-and-boundapproaches, or A* searches that enable finding the essentially minimalsubset without necessarily evaluating all subsets of templates. Inaddition heuristic search methods may be used, such as simulatedannealing, genetic algorithms, and/or random walks in order to quicklyfind subsets with a small number of templates (but not necessarilyminimal). Additionally, optimization algorithms that involve constraintsatisfaction may be used to efficiently find an essentially minimalsubset.

In one embodiment, a goal of a search for a subset of templates thatachieves a certain coverage is to find a subset of templates, whichinvolves a minimal amount of testing time on the user's part. Achievingthe required coverage for the certain user involving the shortesttesting time for the user may be done by estimating time to achieve arequired coverage for the certain user by many templates (e.g., in asearch performed in the space of possible subsets of templates). Andafter evaluating the time it takes to run each of the many templatesand/or subsets of templates, suggesting the certain use utilize a subsetthat includes at least one of the templates, that both achieves therequired coverage and requires an essentially minimal time to complete.In this case, the suggested templates are likely to be templatesrelevant to scenarios that are expected to be used by the certain user,and are likely to be spread to approximately achieve a uniform coverageof the scenarios with the suggested templates.

In some cases, it is useful to refer to a coverage of a test scenariotemplate with respect to a test scenario. In one embodiment, a templateis said to cover a certain test scenario if it involves essentially thesame test steps as the test scenario. Additionally or alternatively,coverage of a template with respects to a test scenario may be thepercentage of test steps of the test scenario that are included in thetemplate. Similarly coverage of a subset of templates with respect to atest scenario may refer to the test steps in the test scenario that areincluded in at least one of the templates in the subset.

Importance of a test scenario for a certain user may also be anobjective that guides the suggestion of a template to the certain user.In one embodiment, test scenario templates that cover a test scenarioare estimated to be important are suggested to the certain user beforeother templates that are estimated to be less important.

There are various ways in which importance of a test scenario may beevaluated. In one example, the importance of a certain test scenario isdetermined by frequency in which the certain test scenario is usedrelative to frequency in which other test scenarios are used. Thus, atest scenario often run by the certain user may be deemed significantlymore important than a test scenario rarely run by the certain user. Inanother example, importance of a certain test scenario is determined bycharacteristics of users that run the certain test scenario, compared tocharacteristics of users that run other test scenarios. For instance, ifa certain test scenario is run primarily by users that hold highpositions in an organization's hierarchy, it is likely that the certaintest scenario is more important than another test scenario that is usedprimarily by users on the lower rungs of the organization's hierarchy.In yet another example, importance of a certain test scenario isdetermined by a financial value associated with the certain testscenario compared to a financial value associated with other testscenarios. For example, a test scenario that involves retaining anunsatisfied customer may be given a high financial value for theorganization compared to another transaction which involves sendingcustomers a notice that their order is still pending.

Profile of a User

A profile of a user may include data related to the user, anorganization to which the user belongs, and/or activity of the user on asoftware systems, such as software systems associated with theorganization. Optionally, at least some of the data in a profile may beconsidered proprietary data. Optionally, the proprietary data mayidentify details regarding the user and/or an organization related tothe profile, such as an organization to which the user belongs. In oneexample, a profile may include proprietary data about the user (e.g.,age, address, languages, skills), and/or proprietary data related to arole of the user in the organization (e.g., job title, placement of userin organizational chart, and/or permissions and/or privileges of theuser). In another example, the profile may include proprietary datarelated to the organization to which the user belongs (e.g., field ofoperation, name and sizes of departments, products and/or servicesprovided by the organization, permissions and/or accounts of theorganization, and/or customers and/or suppliers of the organization). Inyet another example, proprietary data included in a profile of a usermay be indicative of activity of the user. In this example, the activitymay have been previously observed and/or recorded (e.g., by monitoringof the user). Additionally or alternatively, the activity may bepredicted based on characteristics of an organization to which the userbelongs, the position the user holds in the organization, and/or otherattributes of the user (e.g., permissions and/or accounts of the user).

In one embodiment, a profile of a user includes data that is indicativeof test scenarios relevant to the user and/or templates of testscenarios relevant to the user. For example, the profile may includeattributes such as modules used by the user, transactions used by theuser (e.g., identifiers of transactions and the number of times theywere executed), and/or characteristics derived from activity of the user(e.g., accesses to databases, quantities of network data generated,reports generated by the user). Optionally, a profile of a user mayinclude runs of test scenarios of the user, and/or results of a summaryand/or analysis of runs of test scenarios of the user. For example, theprofile may include a list of the transactions that are executed oftenin runs of test scenarios of the user. Optionally, a profile of a usermay include information indicative of test scenario templates utilizedby a user (e.g., templates which the user ran instantiations of).

A profile of a user may be used in order to suggest to the user toutilize a certain test scenario template. Optionally, the template maybe selected from among one or more test scenario templates generatedfrom clusters of runs of test scenarios. For example, given a profilethat indicates that a user performs many transactions involving databaseupdates, the system may recommend for the user a template generated froma cluster that contains runs of test scenarios that involve databaseupdates. In another example, if the profile of a user contains samplesof runs of test scenario run by the user, the system may suggest to theuser to use a template generated from a cluster that contains testscenarios that are similar to a test scenario from which a run in theprofile was instantiated. In yet another example, a profile includesstate information regarding an organization to which a user belongs,such as the time zone and location of a certain site. This informationmay be used to suggest a certain template for a user testing a softwaresystem at the site. For example, a template that involves a report of avacation of a user in advance may be relevant to an organization sitelocated in France, but not for a site in the US (where such a practicemay not be typically required).

In one embodiment, a profile of a user includes permissions of a user.For example, if a profile indicates that a user has permission toapprove vacations, the user may be suggested to run a test scenarioinstantiated from a template that includes such a task. However, if theuser does not have such a permission, then the certain template isirrelevant for the user. In another example, a profile describes aposition of the user in the organization hierarchy. In this example, afirst user may belong to the marketing department, and thus haspermission to enter a new potential client in to the system. A seconduser may belong to the sales department, and thus can send an offer to aclient. Thus, based on the profiles of the users, the system may suggestrelevant templates for the users: to the first user a template thatinvolves entering client data, and to the second user a template thatinvolves making an offer to a client, and not vice versa.

In some embodiments, a profile may belong to an organization. Forexample, it may describe a generic user belonging to the organization.Optionally, a profile of an organization may be utilized to performinitial filtering of test scenario templates for users belonging to theorganization. For example, according to a profile of an organization,the organization may not be authorized to run a test scenarioinstantiated from a certain template (e.g., due to lack of permission).This may make the template irrelevant for each and every user belongingto the organization. However, if the organization is authorized to runan instantiation of the test scenario template, according to the profileof the organization, then a profile of an individual user belonging tothe organization may need to be checked in order to determine if thetemplate is relevant to the individual user (e.g., to check whether theuser has permission to run a test scenario instantiated from thetemplate). In another example, a profile may indicate a method in whichthe organization conducts business. For instance, the profile mayindicate a policy of making payments for goods. Thus, if the profileindicates that the organization always pays after receiving goods, atemplate that describes advanced payment for goods is not relevant forany user in that organization.

In one embodiment, a profile of a user may include usage data of anorganization to which a user belongs. For example, the profile maydescribe which modules are relevant to organization. Based on theprofile, a user will be suggested templates that are relevant to theorganization. Similarly, the profile may describe actions that usersbelonging to the organization may have permission to perform. Forexample, if users of an organization are not authorized to access acertain database, a user belonging to the organization will not beprovided with a template that includes an access to the certaindatabase.

In some embodiments, a profile of a user and/or an organization mayrepresent usage of transactions by the user and/or users belonging tothe organization. Optionally, a profile that represents usage oftransactions may be represented in various ways. The profile may includea list of runs of test scenarios, clusters of runs, and/or transactionsutilized by the user. Optionally, the usage data may be represented as avector in which each dimension may correspond to a certain transaction,cluster of runs, and/or template. In one example, if a user utilized atransaction, a value of the vector in a corresponding dimension is 1,otherwise it is 0. In another example, a value of a dimension in thevector that corresponds to a certain cluster of runs is set according tothe number of runs in the cluster that were run by the user. Thus, themore the user used transactions with corresponding runs in the certaincluster, the higher the corresponding value in the vector.

Having a vector representation for at least some of the values in aprofile makes it easier, in some embodiments, to compare betweenprofiles (e.g., to find similar vectors representing similar profiles ofusers). Additionally, vector representation of profiles may make iteasier to perform mathematical operations, such as vector dot-product ormatrix factorization.

A profile of a user may be utilized to customize a test scenariotemplate for the user. For example, by filtering the test scenariotemplate in order for it to be more appropriate for the user. In oneembodiment, a profile of the user may be used to determine which teststeps, from among the test steps described in a test scenario template,are relevant for the user, and optionally remove test steps that areirrelevant. For example, a profile of the user may indicate that theuser does not confirm shipments. A test scenario template, suggested tothe user may include several test steps that culminate with a test stepinvolving confirming shipment. In such a case, the last test step may beremoved in order to make the test scenario template more appropriate forthe user. In one example, a template may include values related tointernational shipping; however, if the profile of the user indicatesthat the user runs transactions that only involve domestic shipping,certain values concerning international aspects of the transactions maybe removed from the template (e.g., destination country, currencyconversion rates). In another example, a template that includes atransaction that is part of an employee evaluation process may includevalues related managerial assessment of the employee. If the profile ofthe user indicates that the user is not a manager, then those detailsmay be irrelevant for the user. In this case, the user may receive atemplate that only includes transactions in which self assessment of theemployee are performed. In another embodiment, a profile of the user maybe used to determine which values utilized in a test scenario templatemay be appropriate for the user. Optionally, values deemed irrelevant tothe user may be removed from the test scenario template suggested to theuser. Optionally, the user may be requested to provide relevant valuesinstead of the removed irrelevant values. For example, a profile of auser may indicate that the user deals exclusively with domesticcustomers having domestic addresses. However, a test scenario templatemay include default values that correspond to foreign addresses. In sucha case, the foreign addresses may be removed, and the user may berequested to provide examples of domestic addresses.

In one embodiment, a profile of a certain user may be indicative oftransactions run by the certain user, and/or clusters containing runs oftest scenarios run by the certain user that use these transactions.Thus, the profile is indicative of a usage pattern of the user (actualusage and/or expected usage). Optionally, in order to suggest additionaltest scenarios, test scenario templates, and/or clusters for the user,collaborative filtering methods, which rely on usage patterns of otherusers, may be used.

Collaborative filtering is an algorithmic approach generally used formaking automatic predictions (filtering) about the interests of a userby collecting preferences or taste information from many users(collaborating). For example, by detecting similarities between theprofile of the certain user and profiles other users, it may be possibleto suggest to the certain user a template that was utilized by the otherusers, but has not yet been utilized by the certain user. The underlyingassumption is that since the certain user and the other users utilizedsome of the same templates (as evident from the similarity of theirrespective profiles), it is likely that a template that was utilized bythe other users may be useful for the certain user.

There are many algorithmic collaborative filtering approaches that maybe utilized by those skilled in the art to make recommendations for auser based on similarities of a profile of a certain user to profiles ofother users. For example, memory-based methods may be used to select aprofile similar to the profile of the certain user, such as usingnearest-neighbor searches. In another example, model based algorithmsmay rely on the profiles of the other users to generate a model of testscenarios, templates, and/or clusters suitable for the certain user. Themodel generation may utilize many algorithmic approaches such asBayesian networks, latent semantic models, singular value decomposition(and/or other forms of matrix factorization), and/or clustering. Instill another example, a combination of the memory-based and themodel-based collaborative filtering algorithms may be used. In somecases, a hybrid approach that combines memory-based and model-basedapproaches may help overcome shortcomings of the individual approaches.

In one embodiment, a first profile of a first user is considered similarto a second profile of a second user if the first and second profilesindicate that both profiles have a certain number of transactions,clusters, and/or templates in common, and the certain number reaches apredetermined threshold. Additionally or alternatively, the first andsecond profile may be considered similar if the first and secondprofiles have a certain proportion of transactions, clusters, and/ortemplates in common, and the certain proportion reaches a predeterminedthreshold.

In another embodiment, a first profile of a first user is consideredsimilar to a second profile of a second user if a similarity functionapplied to vector representations of the first and the second profilesindicates that the similarity between the vectors reaches apredetermined threshold. For example, the similarity function maycompute the angle (dot product) between the vectors, and if the cosineof the angle is larger than a certain predetermined threshold, theprofiles may be considered similar. In another example, the Pearsoncorrelation may be used to determine the similarity of two vectors. Inyet another example, similarity between binary vectors representingprofiles may be determined according to the Hamming distance of thevectors (e.g., if the hamming distance is below a predeterminedthreshold, the profiles are considered similar).

In one embodiment, profiles utilized by a collaborative filteringalgorithm to suggest templates for a certain user to utilize come fromusers belonging to organizations that are different from an organizationto which the certain user belongs. Often, this can help suggest to thecertain user test scenario templates that may be useful for the certainuser, but are however unknown or not typically utilized within theorganization of the certain user. This may help increase the scope,coverage, and/or diversity of aspects that are tested by runs of testscenarios of the user, which can increase the efficiency of testscenarios, possibly reducing the number of test scenarios that need tobe run.

Cleaning Proprietary Data

In one embodiment, a test scenario template generated from one or moreruns of test scenarios does not include proprietary data captured duringthe stage of monitoring users that ran the test scenarios. Optionally,the one or more runs of test scenarios belong to a cluster and the testscenario template is generated as a representative of the cluster.

In one embodiment, determination of whether certain data, which appearsin a run of a test scenario and/or is obtained from processing data fromthe run, is proprietary data is based on the repetitiveness of the datain runs of test scenarios. For example, if a certain value appears inonly a small proportion of the runs, in runs of a small number of users,and/or runs associated with a small number of organizations, the certainvalue may be considered proprietary. Conversely, if a certain value isthe same in many runs, or in runs of at least a certain proportionand/or number of the users, and/or is associated with at least a certainproportion and/or number of organizations, that value may be consideredto be non-proprietary. Optionally, a predetermined proportion is used asa threshold to determine if a certain value is proprietary or not. Forexample, if a number of different users which ran runs of test scenariosthat included the certain value is less than the predeterminedthreshold, the value is considered proprietary for those users who hadruns that included the value. Otherwise, it may be considerednon-proprietary (since many users had the value in one of their runs).It is to be noted, that “predetermined” refers to both a fixed valueknown a priori (e.g., a threshold of 10 users) and/or a value derivedfrom known logic (e.g., 10% of the users).

In one embodiment, data for which one or more of the following is truemay be considered proprietary data associated with an organizationand/or proprietary data belonging to the organization: the datadescribes an aspect of the organization and/or a user belonging to theorganization; the data appears in a database of the organization; thedata appears in a run of a test scenario associated with theorganization and/or is derived from the run; and/or the data isgenerated by a software system associated with the organization. Forexample, any data on a server belonging to an organization may beconsidered proprietary data associated with the organization. In anotherexample, any data derived from analysis of runs of test scenariosassociated with an organization may be considered proprietary data ofthe organization. Additionally or alternatively, data for which one ormore of the following is true may be considered proprietary data of userand/or proprietary data belonging to the user: the data describes anaspect of a user; the data describes an organization to which the userbelongs; the data appears in a database of the user; and/or the dataappears in a run of a test scenario run by the user.

It is to be noted that as used herein, a phrase like “proprietary data”may refer to proprietary data of an organization and/or proprietary dataof a user. Additionally, phrases like “proprietary values” and“proprietary data” may be used interchangeably in this disclosure.

In one example, proprietary values are removed from a test scenariotemplate generated from a certain cluster (i.e., one or more runsbelonging to the certain cluster were utilized to generate thetemplate). Optionally, the proprietary values are removed by a datacleaner module that operates on the generated template. Additionally oralternatively, removal of proprietary data may be done by other modulesbelonging to the system, such as a template generator, a customizationmodule, a ranking module, and/or a user interface. Optionally, removingthe proprietary values involves selecting a value from the template, andremoving the selected value from the template if the selected valueappears in less than a first predetermined number of runs of testscenarios in the certain cluster. Additionally or alternatively, theselected value may be removed if it appears in runs belonging to thecertain cluster that are associated with less than a secondpredetermined number of different organizations. In this example, boththe first predetermined number and the second predetermined number aregreater than one. Optionally, the first predetermined number and/or thesecond predetermined number are proportional to the number of user withruns belonging to the certain cluster and/or the number of organizationsassociated with runs belonging to the certain cluster. For example, thefirst predetermined number may be set to be the maximum of two and 10%of the users with runs in the certain cluster.

Determining whether data is proprietary may utilize a profile of a userand/or a profile of an organization. For example, any data that appearsin a profile of a user and/or an organization may be consideredproprietary and thus not allowed to be included in a template. Inanother embodiment, the profile may indicate certain data is proprietary(e.g., by placing it a an exclusion list which prohibits utilization ofthe data in templates).

In one embodiment, testing whether certain data is proprietary is doneby querying a database (e.g., a database that contains samples ofproprietary data). Additionally or alternatively, the certain data maybe submitted to a procedure that evaluates the data to determine whetherthe data is likely to be proprietary. For example, the procedure mayperform semantic and/or syntactic analysis of the data to check whetherthe certain data has a certain meaning and/or contains certain patternsthat indicate that it is likely to be proprietary. For example, theprocedure may scan the certain data for strings like “bank account”,“address”, and/or “social security number”.

In another embodiment, a user may provide feedback on certain data whichindicates whether the certain data is proprietary. For example, the usermay review values of data fields prior to running a test scenario andindicate which values the user considers proprietary. Additionally oralternatively, while a test scenario is running, the user may markcertain data as proprietary (e.g., data the user considers should not beseen by the user or other users). Optionally, the user may providefeedback the certain data via a user interface on which the user isrunning the test scenario.

In yet another embodiment, determining whether a certain value may beproprietary may be assisted by noting the source, location, and/or datastructure that contains the certain value. For example, in certainsystems, a value that appears in a checkbox on a screen is likely to bea default value of the system, and thus not likely to be proprietary.However, free text fields on screens are more likely to contain dataentered by a user, and are thus likely to be proprietary. In a anotherexample, in which the certain value appears in a drop-down menu in ascreen, it might be necessary to determine from the context whethervalues in the drop-down menu are proprietary or not. Additionally,certain systems may be built in such a way that makes it relatively easyto determine which data is proprietary and which is not. For example, inSAP ERP meta data, which typically includes general (non-proprietary)data, is clearly marked.

In one embodiment, a value in a test scenario template that isconsidered proprietary is removed from the template. Optionally, theproprietary data is removed by a data cleaner module, a templategenerator module, a customization module, and/or another module and/orcombination of modules. In one example, a value may be deleted from atemplate; consequently, the template may contain an indication ofmissing data (corresponding to the deleted value); prior to running atest scenario based on the template and/or during the run of the testscenario, the missing data needs to be provided (e.g., the user isprompted to provide it). Alternatively, a value in a test scenariotemplate that is considered proprietary may be removed from the templateby replacing it with a default or “dummy” value.

In one embodiment, proprietary data is not included in a template. Forexample, a module generating a template and/or a module customizing atemplate, check whether certain values are proprietary (e.g., bychecking whether a certain flag related to the certain values is raised,or by submitting the certain values to a procedure for evaluation). Ifthe certain values are deemed to be proprietary, they are not includedin the template.

Proprietary data may be removed from runs of test scenarios at differentstages. In one embodiment, the proprietary data is removed from runs oftest scenarios as the runs are recorded and/or identified. For example,a “scrubbed” version of runs, which does not contain certain proprietarydata, may be the data that is provided to components of the system(e.g., a clustering module and/or template generator). Optionally,determining what proprietary data is may relate to general statistics(e.g., utilization of certain elements and/or values by a certainproportion of organizations).

In another embodiment, proprietary data is removed from runs afterclustering of the runs is performed. Optionally, determining what datain the runs should be considered proprietary utilizes the fact thatafter clustering, clusters contain similar runs. In one example, thefact that runs belong to clusters may enable the identification ofcertain values that are shared by many runs (which may indicate that thecertain values are not proprietary), or shared by a few runs (which mayindicate that the certain values are proprietary).

In some embodiments, test scenario templates are generated from one ormore runs of test scenarios belonging to a certain cluster. Optionally,the templates are generated in such a way that they are not likely toinclude data that may be easily traced to specific runs in the certaincluster, users that ran runs belonging to the certain cluster, and/ororganizations associated with runs belonging to the certain cluster. Inthese embodiments, generating a template may involve selecting a valuefrom a run of a test scenario belonging to the certain cluster, checkingwhether the value appears in at least a first predetermined number ofthe runs in the certain cluster; checking whether the value appears inruns in the certain cluster that are associated with at least a secondpredetermined number of different organizations; and if both conditionsare positive, enabling the test scenario template to utilize theselected value. Optionally, if at least one of the conditions isnegative the test scenario template is not allowed to utilize theselected value.

In one embodiment, removing proprietary data from a test scenariotemplate generated from one or more runs of test scenarios involvesremoving most output fields from the runs. Consequently, most of theinformation generated in the runs may be removed. In one example, mostof the information output in the runs is considered proprietary, and istherefore removed. Optionally, as a result of removing most of theoutput fields, the test scenario template does not include expectedvalues for most test steps. Optionally, running an instantiation of thetest scenario template may require a user to provide values in most ofthe test steps.

Customizing a Test Scenario Template

Test scenario templates generated from runs of test scenarios of usersbelonging to different organizations may not be ideal for a certain userfrom a certain organization. For example, the templates may includeinformation such as values that do not suite the certain user and/or thecertain organization, and/or include certain test steps that areirrelevant for the certain user and/or the certain organization. Ittherefore may be beneficial to customize test scenario templates for acertain user and/or a certain organization. Optionally, customizing atest scenario template may be done as part of generating the template(e.g., by the template generator), by another module such as acustomization module, or a combination of modules.

In some embodiments, customizing a test scenario template involvesadding to the test scenario template one or more values that arerelevant to a certain user and/or a certain organization. Optionally,the one or more values that are added replace existing values in thetemplate that may be less appropriate for the certain user and/or thecertain organization. Optionally, at least some of the values added tothe template by the customizing replace proprietary data that has beenremoved from the template (e.g., by a data cleaner module).

In some embodiments, customizing a test scenario template is done usingproprietary data related to a certain user and/or a certainorganization. The proprietary data may be directly related to thecertain user and/or the certain organization, such as internalorganizational data of the certain user and/or the certain organization,and/or data obtained by monitoring the certain user and/or usersbelonging to the certain organization. Additionally or alternatively,the proprietary data may be of a different user and/or differentorganization, and based on similarity to the certain user and/or certainorganization the proprietary data is assumed to be useful forcustomizing the template.

Proprietary data used to customize a test scenario template may havevarious sources. In one embodiment, proprietary data relevant to acertain user, for whom a test scenario template is customized, isobtained by providing the test scenario template to a different user,prompting the different user to provide a missing value, and recordingan input value provided by the different user. Alternatively, the sameprocess may be performed with the certain user, i.e., the test scenariotemplate is provided to the certain user, the certain user is promptedto provide a value, and the value provided by the certain user.

In another embodiment, the proprietary data relevant to the certain userfor whom the test scenario template is customized is obtained by guidinga user to provide the proprietary data while semiautomatically executingan instantiation of the test scenario template. Optionally, the userproviding the proprietary data is the certain user. Providingproprietary data via semiautomatic execution may involve filling a valueinto a field missing a value in an instantiation of the test scenariotemplate. This may be done by selecting a value from a database relatedto the certain user and testing whether the selected value is congruouswith the field. If the selected value is congruous with the field,filling the field with the selected value. Otherwise, prompting thefirst user to provide a value for the field missing a value. Optionally,selecting the value utilizes a heuristics-based algorithm that is run ondata from multiple organizations. For example, the selection of thevalue may be based in part on determining how often the value is used byother organizations. Additionally or alternatively, providingproprietary data via semiautomatic execution may involve filling a valueinto a field missing a value in an instantiation of the test scenariotemplate by identifying users similar to the certain user. Followingthat, selecting a value from runs of test scenarios of the users similarto the certain user and then testing whether the selected value iscongruous with the field. If the selected value is congruous with thefield, the field may be filled with the selected value. Otherwise, thecertain user may be prompted to provide a value for the field missing avalue.

In one embodiment, proprietary data relevant to a certain user, for whoma test scenario template is customized, is obtained by marking locationsof values that were removed from the test scenario template, anddirecting attention of a user to provide data appropriate for the markedlocations. Optionally, the user is the certain user for whom thetemplate is customized. Optionally, the values removed from the testscenario template were removed by a data cleaner module.

One source of proprietary data useful for customizing a test scenariotemplate for a certain user may be a profile of the certain user. Forexample, such a profile may be provided to a module that performscustomization of the template. Data in the profile, such as proprietaryinformation related to the user and/or the organization may be insertedinto the template. For example, a profile of the certain user mayinclude the user's name, address, job title, and/or employee number;these values may be inserted in their appropriate positions in acustomized template in order to save the certain user the time andeffort of inserting them when running an instantiation of the template.In another example, the profile of the certain user may include a listof customers the user works with, parts the user frequently orders,and/or contact information of clients the user frequently interactswith. These values may be automatically inserted into a template inorder to customize it and make it more relevant to the certain user.

In one embodiment, a profile of a user that is used to customize a testscenario template is a profile of a generic user of an organization. Inthis case, customizing the template may utilize information that doesnot identify a specific user. For example, to customize a template thesystem may use activity data from the profile such as a list of modulesthat are frequently run, but not use data such as an employee name of areal employee.

Another source of proprietary data useful for customizing a testscenario template for a certain user may be a data generator related tothe certain user and/or a certain organization to which the certain userbelongs. For example, the data generator may be a program that extractsvalues from a database related to the certain organization. The databasemay hold “real world” information such as actual customer orders, whichcan be used to generate tests that are more relevant.

In one embodiment, customizing a test scenario template for a first userutilizes a customization module and involves obtaining proprietary datarelevant to the first user and substituting a non-empty subset ofproprietary data removed from the template with the obtained data.Optionally, the obtained data undergoes processing prior to beingentered into the template.

In one example, the proprietary data relevant to the first user isobtained from a previously monitored run of test scenario associatedwith the first user; for instance, the run of the test scenario may havebeen run by the first user, run by a user associated to an organizationto which the first user belongs, and/or a user with a similarorganizational role as the first user (e.g., both users are salesmanagers).

In another example, the proprietary data relevant to the first user isobtained from parsing a manual test scenario associated with the firstuser in order to obtain a value associated with the first user. Forexample, the manual test scenario may be a script for running a testthat is intended to be used by the first user, a user associated to anorganization to which the first user belongs, and/or a user with asimilar organizational role as the first user. After obtaining the valueassociated with the first user, a non-empty subset of the removedproprietary data from the template is substituted with the obtainedvalue.

In yet another example, the proprietary data relevant to the first useris obtained from analyzing a database of the software system associatedwith the first user to obtain a value associated with the first user.The database may include data on and/or be accessed by the first user, auser associated to an organization to which the first user belongs,and/or a user with a similar organizational role as the first user.After obtaining the value associated with the first user, a non-emptysubset of the removed proprietary data from the template is substitutedwith the obtained value.

In one embodiment, the customization module is also configured to:provide the customized test scenario template to a second user, promptthe second user to provide a missing value, and record an input valueprovided by the second user. Optionally, the first user and the seconduser are the same user.

Values used for customization of templates may be obtained, in someembodiments, by exploiting wisdom of the crowd. This “wisdom” may begained by analyzing runs of users from different organization in orderto find certain patterns and common values. In one example, manyorganizations have a dummy customer entry for testing; for example, acustomer named “test”. Thus, entering “test” as a user is likely toallow a test scenario to run, so a customization module may try usingthe customer “test” in a template being customized. In another example,SAP catalog numbers usually start from 1000, and 1000 usually describesa catalog number for tests and not a real number. Thus, when customizinga template, if a catalog number is needed, the value 1000 may be tried.In both of these examples, the knowledge of which default values may begained by examining runs of users from different organizations. Thisallows a customization module to discover from the crowd certain valuesthat may not be apparent to whomever is customizing the template.

In embodiments described in this disclosure, after generating a testscenario template from one or more runs of test scenarios (e.g., using atemplate generator), the template might undergo further processing suchas removal of proprietary data, customization, ranking, and/or thetemplate may be suggested to the user. Optionally, the one or more runsof test scenarios belong to a certain cluster of similar runs of testscenarios. FIG. 9A to FIG. 9G illustrate some, but not all, combinationsof system modules that may be used in embodiments described in thisdisclosure to process the template and/or suggest the template to auser.

FIG. 9A illustrates a combination of system modules that may be used inembodiments in which a template generator 770 generates a test scenariotemplate from one or more runs of test scenarios. The template isprovided to a ranking module 771 that may rank the template, forexample, by assigning it a score proportional to its relevancy to acertain user. Optionally, the ranking module 771 receives a profile of acertain user and/or a certain organization and utilizes data from theprofile, in order to rank the template. Following that, a data cleaner772 receives the template and may remove proprietary data from thetemplate. Optionally, the data cleaner 772 receives a profile related toa certain user and/or a certain organization, and utilizes the profileto remove certain proprietary data from the template. After removingproprietary data from the template, the template from which proprietarywas removed is provided to a customization module 773 that customizesthe template by adding certain data to the template. Optionally, thecustomization module 773 receives a profile of a certain user and/or acertain organization and adds data from the profile, which relevant tothe certain user and/or organization, to the template. The customizedtemplate from which proprietary data was removed is then provided to auser interface 774. Optionally, the user interface 774 presents to auser a suggestion to run an instantiation of the customized templatefrom which proprietary data was first removed.

It is to be noted that in this disclosure, though modules may beillustrated in figures as separate elements, in some embodiments, theymay be implemented as a single element that performs the functionalityof modules. For example, the software module that implements thetemplate generator 770 may also perform the tasks of the data cleaner772. In another example, the customization module 773 and the datacleaner 772 are realized by the same software programs.

In other embodiments, after being processed by various modules, atemplate generated by the template generator need not be provided to auser interface. For example, it may be stored in the system for furtheruse. FIG. 9B illustrates a similar portion of a system to the systemillustrated in FIG. 9A, however in this system after being subjected toranking, removal of proprietary data, and customization, a template isnot presented to a user via a user interface.

The order in which a template may undergo processing by system modulesmay differ between embodiments. For example, FIG. 9C illustrates acombination of system modules which includes the template generator 770which provides a template for customization by a customization module775. Following the customization, a data cleaner 776 removes proprietarydata from the template. The template is then provided to a userinterface 777, for example, in order to be suggested to a user. FIG. 9Dillustrates a similar system, however in it a template generated by thetemplate generator 770 is first provided to a data cleaner 778. Afterremoving proprietary data from the template, the template is provided toa customization module 779. The customized template may then bepresented to a user via the user interface 777.

In some embodiments, a user interface is not necessarily used to suggesttemplates to a user; it may be utilized by other system modules toperform their task. FIG. 9E illustrates a combination of system modulesthat may be utilized in embodiments, in which a ranking module 782 and adata cleaner 780 interact with a user interface 781. For example, thedata cleaner 780 may present a template generated by the templategenerator 770 to a user on the user interface 781, in order for the userto mark and/or approve certain proprietary data the data cleaner foundin the template. Additionally, the ranking module 782 may present aranking of the template in order for the user to verify the rankingand/or edit it. Optionally, the user may then determine what is to bedone with the template (e.g., should the user run it, save it for later,and/or discard it). FIG. 9F illustrates a combination of system modulesthat may be utilized in embodiments, in which a customization module 775utilizes a user interface to perform customization of a template. Forexample, data that was added to a template generated by the templategenerator 770 is presented to a user on a user interface 784 forapproval and/or editing. Following that, the template may be subjectedto removal of proprietary data by a data cleaner 785. For example, theproprietary data may be added by the user via the user interface 784.

In some embodiments, a template that is generated by a templategenerator does not undergo additional processing by system modules. Forexample, as illustrated in FIG. 9G, after being generated by thetemplate generator 770, the template may be sent to the user interface774 (e.g., to be suggested to a user).

While some of the above embodiments may be described in the generalcontext of program components that execute in conjunction with anapplication program that runs on an operating system on a computer,which may be a personal computer, those skilled in the art willrecognize that aspects may also be implemented in combination with otherprogram components. Program components may include routines, programs,modules, data structures, and other types of structures that performparticular tasks or implement particular abstract data types. Moreover,the embodiments may be practiced with other computer systemconfigurations, such as: cloud computing, a client-server model, gridcomputing, peer-to-peer, hand-held devices, multiprocessor systems,microprocessor-based systems, programmable consumer electronics,minicomputers, and/or mainframe computers. The embodiments may also bepracticed in a distributed computing environment where tasks areperformed by remote processing devices that are linked through acommunication network. In a distributed computing environment, programcomponents may be located in both local and remote computing and/orstorage devices. Some of the embodiments may also be practiced in theform of a service, such as infrastructure as a service (IaaS), platformas a service (PaaS), software as a service (SaaS), and/or network as aservice (NaaS).

Embodiments may be implemented as a computer implemented method, acomputer system, and/or as a non-transitory computer-readable medium.The non-transitory computer-readable medium comprises program code whichprovides, or participates in providing, instructions to a processor. Thenon-transitory computer-readable medium may be implemented, for example,via one or more of a volatile computer memory, a non-volatile memory, ahard drive, a flash drive, a magnetic data storage, an optical datastorage, and/or any other type of a tangible computer memory to beinvented that is not transitory signals per se. The program code may beupdated and/or downloaded to the non-transitory computer-readable mediumvia a communication network such as the Internet.

Herein, a predetermined value, such as a predetermined threshold, may bea fixed value and/or a value determined any time before performing acalculation that compares a certain value with the predetermined value.A value may also be considered to be a predetermined value when thelogic, used to determine whether a threshold that utilizes the value isreached, is known before start of performing computations to determinewhether the threshold is reached.

In this description, references to “one embodiment” mean that thefeature being referred to may be included in at least one embodiment ofthe invention. Moreover, separate references to “one embodiment” or“some embodiments” in this description do not necessarily refer to thesame embodiment. Additionally, references to “one embodiment” and“another embodiment” may not necessarily refer to different embodiments,but may be terms used, at times, to illustrate different aspects of anembodiment.

The embodiments of the invention may include any variety of combinationsand/or integrations of the features of the embodiments described herein.Although some embodiments may depict serial operations, the embodimentsmay perform certain operations in parallel and/or in different ordersfrom those depicted. Moreover, the use of repeated reference numeralsand/or letters in the text and/or drawings is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various embodiments and/or configurations discussed. Theembodiments are not limited in their applications to the details of theorder or sequence of steps of operation of methods, or to details ofimplementation of devices, set in the description, drawings, orexamples. Moreover, individual blocks illustrated in the figures may befunctional in nature and therefore may not necessarily correspond todiscrete hardware elements.

While the methods disclosed herein have been described and shown withreference to particular steps performed in a particular order, it isunderstood that these steps may be combined, sub-divided, and/orreordered to form an equivalent method without departing from theteachings of the embodiments. Accordingly, unless specifically indicatedherein, the order and grouping of the steps is not a limitation of theembodiments. Furthermore, methods and mechanisms of the embodiments willsometimes be described in singular form for clarity. However, someembodiments may include multiple iterations of a method or multipleinstantiations of a mechanism unless noted otherwise. For example, whena processor is disclosed in one embodiment, the scope of the embodimentis intended to also cover the use of multiple processors. Certainfeatures of the embodiments, which may have been, for clarity, describedin the context of separate embodiments, may also be provided in variouscombinations in a single embodiment. Conversely, various features of theembodiments, which may have been, for brevity, described in the contextof a single embodiment, may also be provided separately or in anysuitable sub-combination. Embodiments described in conjunction withspecific examples are presented by way of example, and not limitation.Moreover, it is evident that many alternatives, modifications, andvariations will be apparent to those skilled in the art. It is to beunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the embodiments.Accordingly, this disclosure is intended to embrace all suchalternatives, modifications, and variations that fall within the spiritand scope of the appended claims and their equivalents.

What is claimed is:
 1. A computer system comprising: a test identifierconfigured to identify runs of test scenarios run by users belonging todifferent organizations on software systems associated with thedifferent organizations; a route analyzer configured to receive a run ofa test scenario and to identify a route that the run follows; a databaseconfigured to store the route; a route retriever configured to receive atemplate route belonging to a test scenario template and to retrievefrom the database divergent routes that diverge from the template route;wherein the test scenario template is associated with a certainorganization that does not belong to the different organizations; asubset selector configured to select from the divergent routes a subsetof divergent routes that are similar to each other; and a templatemanipulator configured to identify that size of the subset reaches apredetermined threshold greater than one, and to manipulate the testscenario template according to the subset.
 2. The computer system ofclaim 1, wherein the template manipulator is further configured tomanipulate the test scenario template by updating the test scenariotemplate according to one or more of the routes belonging to the subset.3. The computer system of claim 1, wherein the template manipulatorfurther configured to manipulate the test scenario template bygenerating a new test scenario template based on the test scenariotemplate and one or of the routes belonging to the subset.
 4. Thecomputer system of claim 1, wherein a route defines a sequence of one ormore elements involved in running a test scenario; wherein an element isselected from the group consisting of test steps, transactions, andscreens.
 5. The computer system of claim 4, wherein a divergent routethat diverges from a template route is characterized by a differentsequence of elements compared to sequence of elements defined by thetemplate route.
 6. The computer system of claim 4, wherein the divergentroutes that are similar to each other are characterized by at least oneof the following features: the divergent routes start from and end withsame screens, the divergent routes have a same sequence of screens butwith different user keys, and the divergent routes comprise same changesin relation to the template route.
 7. The computer system of claim 1,wherein the predetermined threshold is greater than three divergentroutes belonging to three different organizations, and the templatemanipulator is further configured to identify that size of the subsetdoes not reach the predetermined threshold and not to manipulate thetest scenario template according to the subset.
 8. The computer systemof claim 1, wherein the route retriever is further configured toidentify divergent routes that diverge from the template route and lateron converge back to the template route.
 9. The computer system of claim1, wherein the route retriever is further configured to identifydivergent routes that diverge from the template route and do notconverge back to the template route.
 10. The computer system of claim 1,wherein the route retriever is further configured to identify divergentroutes that merge with the template route.
 11. The computer system ofclaim 1, wherein the predetermined threshold is selected such thatreaching the predetermined threshold indicates that the subset is likelyto be useful for other organizations.
 12. The computer system of claim1, further comprising a monitoring module configured to monitor theusers running the test scenarios on the software systems that belong tothe different organizations and to provide data obtained from themonitoring to the test identifier; and further comprising a rankingmodule configured to rank the manipulated test scenario templateaccording to the size of the subset.
 13. The computer system of claim 1,further comprising a ranking module configured to rank the manipulatedtest scenario template according to number of different organizationsassociated with the routes belonging to the subset.
 14. A computerimplemented method comprising: identifying runs of test scenarios run byusers belonging to different organizations on software systemsassociated with the different organizations; receiving a run of a testscenario; identifying a route that the run follows; storing the route ina database; receiving a template route belonging to a test scenariotemplate; wherein the test scenario template is associated with acertain organization that does not belong to the differentorganizations; retrieving from the database divergent routes thatdiverge from the template route; selecting from the divergent routes asubset of divergent routes that are similar to each other; andidentifying that size of the subset reaches a predetermined threshold(greater than one) and manipulating the test scenario template accordingto the subset.
 15. The computer implemented method of claim 14, furthercomprising manipulating the test scenario template according to thesubset by updating the test scenario template according to one or moredivergent routes belonging to the subset.
 16. The computer implementedmethod of claim 14, further comprising monitoring the users running thetest scenarios on the software systems that belong to differentorganizations and providing data obtained from the monitoring for use inthe identifying of the runs of test scenarios.
 17. A non-transitorycomputer-readable medium for use in a computer, the computer comprises aprocessor, and the non-transitory computer-readable medium comprising:program code for identifying runs of test scenarios run by usersbelonging to different organizations on software systems associated withthe different organizations; program code for receiving a run of a testscenario; program code for identifying a route that the run follows;program code for storing the route in a database; program code forreceiving a template route belonging to a test scenario template;wherein the test scenario template is associated with a certainorganization that does not belong to the different organizations;program code for retrieving from the database divergent routes thatdiverge from the template route; program code for selecting from thedivergent routes a subset of divergent routes that are similar to eachother; and program code for identifying that size of the subset reachesa predetermined threshold (greater than one) and manipulating the testscenario template according to the subset.
 18. The non-transitorycomputer-readable medium of claim 17, further comprising program codefor manipulating the test scenario template according to the subset byupdating the test scenario template according to one or more of thedivergent routes belonging to the subset.
 19. The non-transitorycomputer-readable medium of claim 17, further comprising program codefor manipulating the test scenario template according to the subset bygenerating a new test scenario template based on the test scenariotemplate and one or more of the divergent routes belonging to thesubset.
 20. The non-transitory computer-readable medium of claim 17,further comprising program code for determining that the size of thesubset does not reach the predetermined threshold and refraining frommanipulating the test scenario template according to the subset; wherebynot reaching the predetermined threshold indicates that the divergentroutes belonging to the subset are not likely to be a suitablealternative to the template route for other organizations.